We recently had an interesting situation with our test suite while working on a Rails project. When our entire test suite was run (rspec -> cucumber -> jasmine) we would have all tests pass, however, when cucumber was run on its own it would fail. Upon inspecting our stack trace we saw that a followed redirect in cucumber was not rendering correctly. This meant that the environments differed between running the entire suite and running the acceptance tests in isolation. How could this be, we all thought, an app is an app and the environment is test no matter how it is run.
Upon inspection we were able to follow the problem to a 'singleton' we had within the system. We had a mailing list object that was meant to have a single instance to represent the single mailing list we are keeping. Within our seeds.rb file we had a check to see if an instance exists, and if not to then create an instance. Our class looked like this:
All developers on the project understood that mailing list was to be called by mailing_list.instance and in our heads we thought of it as a singleton. In production this was working. Thankfully, we had tests to expose the problem with our logic. When rspec ran it created it's first MailingList instance at id 1, which made instance work throughout the test suite and explained why our tests passed when the entire suite was ran. However, cucumber deleted its instances of MailingList after a scenario had finished. This was a problem because the mysql backend kept issuing ids in order. Therefore, the above code would return a nil object when we asked for the instance.
Our revised singleton looks a little better:
We now give instance more importance. We were able to drop the singleton creation out of our seeds file since calling instance always ensures a singleton. When we use create it gives an informative error and leads the developer to use instance, as intended. Moral of the story, tests are more important than you think.
