Too Much Time Wasted Running Tests

When I arrived at my current project, it had a large stack of integration tests using a combination of Spring, Hibernate, DbUnit and Unitils. The test suite was backed by MySql in all environments. (local, CI and Production.) The problem with this solution is time! It takes anywhere from 3 to 4 minutes to execute mvn clean test on our dev boxes. Now, that does sound like a long time to wait, and I've worked on projects that have taken over 15 minutes to complete the test suite, but it adds up. Multiple that time by the number of times a day a developer runs the test suite, times that by the number of developers on the team, and you start to get serious amounts of time spinning your wheels. I conservatively estimated the team as a whole was waiting over 3 hours a day for the test suite to complete. Beyond these numbers, we use TDD and it is good development practice to check in as often as you can. If the time taken to run your test suite hinders that, you are already slipping into the dark. I set about finding a solution; looking around, asking some questions and some scratching of heads these two options were selected.

• Split off the long running integration tests and have them only run under a specific maven profile
• Use a faster in-memory database like HSQLDB or H2

Splitting off the longer integration tests does allow the developer to run their tests quicker, but they have to wait for CI to catch any integration test failings. This means if there are errors with integration that code as been checked in. Checked in code that has errors...err. no! Changing to an in-memory database means your schema definition has to change because it will inevitably contain database specific elements. Furthermore, you are no longer using the same database as your production environment, and therefore more testing will have to be done to ensure there are no gotchas when deploying to production due to database change. Neither option sat well with us, so we opted for a combination. Run all tests against an in-memory database in development, but have CI run against a production like database. With this option you get the fast, full test suite locally, and have CI deal with the longer wait that comes with using a production like database. Of course, there is the chance that your CI will still fail, but the likelihood is much lower. How this was actually achieved will be in my next post. This will include how I overcame schema difference without duplication; dialect issues and handling return type class issues from different database drivers.