The company I’m currently consulting to are pretty good when it comes to their internal development systems, although one of the main areas I can see for improvement is in continuous integration. They currently have a full build process is manually initiated, and works pretty well. I really think they need to go to the next step, that of continuous integration.
A couple of days ago I was checking in some very subtle but serious changes to the way we accessed SQL Server (we switched to use User Instances so that we could run as an LUA user). We were all a bit nervous about this change because although it was a relatively minor change, it was a significant unknown, in fact I was the only one who had any idea of how user instances worked. So as part of the code review, the reviewer asked me to run all of the unit tests just to be sure we hadn’t inadvertently broken anything, and sure enough a whole heap of them just curled up and died a horrible death. Being the insecure person that I am I immediately rushed to the conclusion that it was my changes and started questioning my own understanding of user instances. When running the application, I was able to reproduce the error under certain conditions on my machine, but no-one else was able to reproduce the error on their machine. The next day I spent the best part of the morning looking at the problem and eventually discovered that firstly the reason I could reproduce it on my machine was because I was using an obscure (non-default) option in the application, and secondly that the bug occurred with or without my changes from the night before. Armed with this information I then went to the team lead who then spent the next half of the day going back through labels in our source control attempting to find the release that this bug first occurred in. He eventually found it to be due to a check-in about 4 or 5 days prior. This indicated 2 things, firstly people had not been running even the effected unit tests for their check-ins, and secondly if we had continuous integration, and unit tests were run automatically at the point of check-in, we would not have wasted a day and a half of development time tracking down the origin of the bug. I told the development manager that this incident re-affirmed my faith as a unit-test evangelist.