This will allow us to have global setup and teardown for tests.

For example, we can automatically clear caches after each test, to avoid
annoying intermittent failures like #1879 and #1886.