Quality is Not Accidental

I've had plenty of opportunities to manage releases of products in my career. As I mentioned in an earlier post, most of them were large mission-critical enterprise applications. Open source projects are different beasts entirely, but they don't have to be. A little bit of Enterprise rigor combined with the vast resources of the community can reap great rewards.

Lately the Maven project has been taking a lot of heat from various sources about stability and over-all quality. For the most part, they were right. The Maven team is very strong and certainly no one intends for these problems to happen, but ultimately they were.

In a well run commercial project, you have a team of QA engineers dedicated to validating fixes and checking for regressions. Unfortunately no matter how hard you test and identify regressions, business pressures often force a release to occur before it's truly ready.

In an OSS project, the team is typically made up of developers and not professional testers (or docs writers). As such, you tend to end up with lots of code and less than stellar testing coverage. However, if you expand the concept of the team a bit to include the user community at large (aka The Community), you can easily smoke the commercial equivalent for a couple of reasons:

• The user community is comprised of a nearly limitless number of expert users who are highly motivated to check for regressions
• Each user in the community brings a slightly different set of use cases to the testing effort
• The absence of business pressure for a release can be a blessing if harnessed correctly.

During a review of the open issues while planning 2.0.10, I became aware that a significant number of open issues start with "this used to work until 2.0.[x]". I became frustrated and a little embarrassed to realize how bad of a regression problem we had going on.

At that point I decided that my personal priority for 2.0.9 had to be No more regressions.

The normal release process for Maven is to stage a release, email the dev list and wait for votes or show stopper issues to occur. The norm for most releases is 72 hours, but with Maven core releases it was common to let it bake for a week or more. Based on history, I was positive that the first few attempts wouldn't make it through, so we started with a "pre vote" instead of a vote email.

It seemed that each "pre vote" staged release we posted for dev list testing showed yet another how come no one noticed that? regression. It became apparent that we needed more than ever to harness the power of the full community to squash these regressions. Since tossing out multiple versions all called "2.0.9" to such a wide audience was clearly a bad idea, we started appending -RC[x] to distinguish them. Additionally, we needed to have a set of operating parameters to guide this broad level of testing, lest we have chaos in the flood of bug fix requests.

The gist of the operating parameters was:

• We won't fix more issues unless it's a demonstrated regression from the previous version. (2.0.8)
• We will fix all regressions identified unless fixing it is riskier than leaving it.
• All changes will be accompanied with a core Integration Test.
• Community participation will drive the quality of this release.
• We will continue this progress as long as it takes.

That last item relates specifically to the lack of business pressures to force a premature release. Being OSS allows us to harness the immense power of the community for as long as it takes to get the release right.

So far we've gone through 8 release candidates, and, as a team, we were able to correct every regression identified, something I wasn't sure would be possible. The official release is now staged awaiting a last check and formal vote. If all goes well, the release will be out later this week.

The feedback received on the new process has been overwhelmingly positive and the level of testing by users was surprisingly high for the first try. A special thanks to all of you that tested and provided feedback, even if it was just confirming that everything looked good. Several users went through the effort to validate all eight of the RCs on corporate CI systems and that's no small feat.

Going forward, I'm hoping to be able to put out more regular core releases and by holding the No regression mantra from the beginning, future releases should be easier. As promised the lessons learned from this release will be codified into a new formal release procedure. I'm currently toying with introducing the idea of milestone releases for regression testing should any future release go beyond a month or so since the last release.

Only time will tell if 2.0.9 stands out from all the predecessors in terms of quality, but it feels like we're on the cusp of a new era. WDYT?