Static State

Beware of static state. Need I say more?

Ok, I guess I shall, regardless of if I need to. If your automated tests all run from the same context (in my case, a JVM instance), static state will bite you... eventually. You may think you are dealing with the state just fine, but it will hurt you eventually.

I have seen many problems crop up with static state so far. Cached data from the database causes data from one test to bleed into another test. Between tests, you will likely have different data. If the data is not based on a fixed set, it is inevitable that some of the data will be slightly tweaked between tests. If that data is cached, the later test will end up with a cached copy of the earlier test's data. This may not crop up right away, and can happen at any time with cached data. A new test that runs before some old tests could cause the old tests to fail. If the order of tests is not predictable in your automated build, you could get a new ordering that sets up a perfect storm to cause some invalid cached data to stick around and haunt another test.

Similarly, you might have some mock versions of static data that makes testing easier. If you don't clean up the mock data, another test may be expecting the default behavior, and again you are failing because of an indirect cause.

Ultimately, static data can be useful and convenient at times, just be aware that your test environment gets weakened with every static cache or storage you use. You can't always simply try to clean up the cached data when you use it. Consider a class, A, that caches some form of data. Your current test depends on class B, which luckily doesn't use the cached data in A, but it does use class C. You don't deal with cleaning up A. Then, down the road, someone changes C to update or view the static state in A. All of a sudden the tests for B are dependent on static state you had no idea about. Or you are modifying that data unknowingly, and affecting all tests that run after yours that depend on the state of A being in a default state.

I've found the most effective approach is a setup or tear down method in your tests that explicitly clears all known cache. This will ensure no test can hide data that leaks into another test, causing an unexpected (and difficult to track) test failure.

Just be on the lookout for the telltale sign of static state causing failures. If a test class fails on the build machine (or in a group), but it doesn't when run individually... you might be seeing static state showing its ugly head.