Beta tests, a double edged sword

A lot of software companies have a beta test phase for their software before they go to commercial release. It appears to me that there are 2 main reasons companies release beta's of their software.

Firstly the desire to make the product more stable in the real world by getting people in the real world, (not QA staff back in the controlled test environment), to use the software in a way that as close as possible resembles the way in which the software will be used when it is released commercially. This is very valuable, I have seen so many times bugs turn up in real world situations that no matter how much testing you get internal QA staff to do, you would be highly unlikely to see back at the office.

The second reason is to get public exposure of the software before the release date. The idea is to create a certain amount of excitement about the impending release of the product, get potential end users to start using the software for a limited trial period, let them see the benefits that the software will offer, and hopefully have a list of potential customers at the end of the beta phase that are ready to fork out money for the product when it is released.

It is possible that these 2 objectives can easily conflict with each other. By that I mean if you release a beta that has a lot of bugs, then the excitement you are trying to generate will be offset by the mistrust in the minds of the beta testing community that you as are not capable of delivering stable software. On the other hand, if you delay your beta test until the software is perfect, than having a 30 day beta cycle that returns very few bugs is just going to delay getting the commercial release out and consequently delay getting returns on your investment.

A balance needs to be struck and I think there are a few simple rules that need to be followed to ensure that you get the most out of your beta test phase.

1. Beta testing does NOT replace a good internal quality assurance process. Anyone not convinced of the value of a good QA process needs to read Joel Spolsky's rant on testing. My thoughts on the subject are that before a beta goes out, there should be a series of tests that ensure the software is suitable for public consumption. Basic things like it installs and uninstalls fine on targeted systems, There are no serious bugs in the day to day expected use of the software, obvious spelling and grammatical errors will reflect poorly on your companies professionalism, etc... The exact requirements will vary depending on what type of software you're releasing, and the target market.

2. The Beta test phase should be well managed. Someone, ideally from the QA team, should be made the single point of contact for all beta testers, and there should be some kind of forum for beta testers to develop a relationship with this person, say a blog for instance. beta testers should be encouraged to use this blog for reporting bugs, giving feedback, asking question etc.... This should be a place where the person in charge of the process is actively answering questions, dealing with issues, thanking people for input and generally presenting a human face to the whole process.

3. Issues need to be resolved quickly. If there are any serious issues, they need to be dealt with as fast as possible, and a new beta posted within days of the issue being reported. I don't think I need to explain the importance of this rule, it I obvious that perception is everything.

4. Beta testers should be rewarded. These people are essentially doing free work for you, there should be some sort of incentive at the end to show gratitude for there commitment. I would suggest a small discount on the final product would be adequate incentive, and would encourage beta testers to sign up for the next beta process.

5. The beta test needs to be of a pre-determined length. I would say aim at between a 30 and 60 day beta phase, if you go on too long then you lose some of the initial excitement you created when you started the beta test phase. There are exceptions to this rule, and it depends on the size of the application you're developing, and the size of the beta test community, and also if one of your objectives is to get customer feedback about what features they want, then 60 days will probably be too short a time to get that feedback, feed it back through the relevant channels and back into the development cycle.

Beta tests can achieve the goals of creating more reliable software as well as a community of people excited about your software, however, it can also create a distrust of your software and your company if the process is not managed correctly.