OpenDS: Open Source Community Development Improves Product Quality

Gary Williams, a staff engineer and the QA lead of OpenDS,
published a great article with Marina Sum on the topic of how working
on an open source software project has improved quality in product
development.  The process is without challenges which he outlines in
the article as well.  However, he also gives great detail about the
test harness that is used, the amount of automation and community
involvement to address the challenges and get high quality product in
community hands more frequently.  The full article
is available on the Sun Developer Network here

These
are the types of processes that quality open source projects do as a
part of the project development process.  Indira Thangasamy, produced a similar
article on how they approach QA within the OpenSSO project. 
As companies evaluate other open source projects, especially in these
challenging economic times where cost reduction provide stronger
rational’s to consider starting projects using open source software. 
The quality approach of communities becomes an important differntiator
as companies use open source in production and customer facing systems.

Here is a quick overview of the test harness used on OpenDS:

We use open-source, Java platform-based test tools, such as the
following, not only to demonstrate our support for open source but also
to ensure that they are accessible to everyone:

Here are a couple of other highlights:

  • Unit Testing and Automation:  "Testing starts in the programming phase with unit tests, which verify
    that the code works as intended and which must exist for all features.
    Today, we run 30,000 automated unit tests daily on different Java
    virtual machines. No code can be integrated without satisfying the
    precommit requirements."
  • Code coverage — With open-source EMMA, we find out the
    number of code lines, blocks, methods, and classes that are exposed by
    the unit and functional tests. Part of that information pinpoints the
    amount of the code tested as a percentage of the total, defining if
    we’ve met the quality criteria. We also define which areas of the code
    are not tested, called coverage holes, and create new tests to fill
    them.
  • Feature coverage — OpenDS delivers features that customers
    want, that is, customer requirements. Each feature is recorded as an
    issue in the Issue Tracker, a tool that monitors defects. This data
    tells us the state the features are in and their status: Ready for Test
    or Tested.
  • Documentation coverage — To ensure that the documentation is
    reviewed according to the test plan, we adopt a two-phase documentation
    review process: a technical review of the content followed by a formal
    QA review. Like the product features, the documentation is divided into
    categories—books, chapters, and sections—that are recorded in the Issue
    tracker. Through this coverage, we measure the percentage of the
    documentation reviewed over time and identify the reviewers and review
    status.
  • Defect rates — This is a traditional measure. The goal is to
    have no high-priority bugs open at release time. Our Bug Council
    constantly studies the defects and assesses the risks to customers. We
    also plot simple graph trends to gauge how well the project is
    converging.

Thanks to Gary and Marina for publishing this article and allowing the community to learn from your experience. 

Advertisements
This entry was posted in Sun. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s