|
By Gary Williams, with contributions from Marina Sum, January 8, 2009
|
|
|
|
Quality is indisputably the number-one requirement for any product or service. It's been rightly said that when you're out of quality, you're out of business. Superior quality, however, is never an accident and achieving it takes the concerted efforts of creative, competent, and dedicated professionals in both the development and testing arenas.
The advent of open-source software over the past years has presented both opportunities and challenges for test engineering. As the lead test engineer for OpenDS, I'd like to share with you the nuances of testing open-source software and, in particular, OpenDS.
|
OpenDS is a Sun-sponsored, open-source directory-server project for developing a free and comprehensive next-generation directory service. Written entirely in the Java programming language according to the Lightweight Directory Access Protocol (LDAP) and Directory Services Markup Language (DSML) standards, OpenDS shipped its 1.0.0 release in August 2008. Release 1.2.0 is scheduled for late January 2009, slated to be integrated into the next OpenSolaris release.
|
|
|
Opportunities, Challenges, and Roles
Openness and transparency mean that software is exposed to scrutiny by the community, with issues often detected early in a project's life cycle. Concurrently, contributors are motivated—and even pressured—to deliver software that meets the community's needs.
However, several challenges have emerged, resulting in unique and often rewarding and dynamic roles for test engineers:
- The fast growth of open-source software has led to the involvement of many contributors—developers, marketers, documentation authors, users—who comprise a large extended team. Quality has effectively become the entire team's responsibility. Nonetheless, the testing team is still accountable for quality. We must exchange and coordinate a vast amount of information, communications, and test results with all concerned, then digest and assimilate all of it.
Make no mistake, interest and involvement from the community directly affect the quality of the software. It's a huge plus to have members who frequently test, evaluate, and improve the code, and who also reject enhancements that introduce too many bugs. As a result, quality and success of the community are interlinked.
- Simultaneously, we must be open and transparent—just like the software. We are at the "coal face," working side by side with a broad spectrum of individuals, both internally and externally.
- We must continually make intelligent choices on what and what not to test. Of paramount importance are what might affect customers' businesses and release schedules and what actions to initiate to minimize the risks.
- Another key role that we assume is that of an investigator who must research and present quality-related information for the stakeholders, in this case, the community. Ultimately, that data serves as the basis for risk-related decisions.
To summarize, it's a constant balancing act to build a community around open-source testing and to sign off a world-class product for timely delivery. But we don't do it all by ourselves: I can't repeat enough that it's a team effort mounted by like-minded individuals.
Guidelines and Practices
The OpenDS QA team consists of about 10 members, all located in Grenoble, France. We believe that quality must be built into the product from the start and, in close collaboration with our developer counterparts, we do our best to head off emergence of bugs through testing subsequent to code integration. Realistically, the later we find bugs, the more costly and cumbersome it is to fix them.
In concert with the community over the years, we have defined rules, guidelines, and best practices for testing OpenDS. Here are a few examples:
- We use open-source, Java platform-based test tools, such as the following, not only to demonstrate our support for open source but also to ensure that they are accessible to everyone:
- Testing starts in the programming phase with unit tests, which verify that the code works as intended and which must exist for all features. Today, we run 30,000 automated unit tests daily on different Java virtual machines. No code can be integrated without satisfying the precommit requirements.
We also run automated, daily acceptance tests so that community members can quickly reap a return on their contributions and promptly react. That's a classic "two-way street."
- Each feature is explored and tested against a user scenario by a test engineer, who also verifies that the feature's behavior matches the related documentation. Every day, we download OpenDS and the DSML gateway and run 2,000 functional regression tests against the build on different platforms.
- Success of the unit and functional tests proves that the basic building blocks are solid, freeing us to focus on system or scenario testing. That's where "we test software like customers use it"—an all-important phase. Peter Drucker once made an excellent point: "Quality in a product or service is not what the supplier puts in. It is what the customer gets out and is willing to pay for."
We always hunger for real-life situations and constantly solicit input from the community. Occasionally, we also arrange on-site visits to stay in close touch with customers and to brainstorm with them on their current plans for OpenDS and wish lists for the future. Those powwows are invaluable and gratifying for both sides.
- To attract adoption and help customers maintain quality deployments of open-source software, we insist on accurate, clear, and complete documentation and an efficient, simple installation process. Consequently, many of our tasks are oriented toward making that a reality.
- We perform validation for LDAP conformance, security, and compliance. Adherence to the standards is a must.
In addition, regular quality checkpoints are in place to ensure a high standard for the OpenDS builds and their availability. Besides daily builds, we schedule promoted builds (every few weeks), milestone builds (at regular project intervals), and release candidates, each guaranteeing predefined levels of quality. All those builds are available to the community for further testing.
Goals and Measurements
Our main goal is to deliver a quality product on schedule. As widely acclaimed as OpenDS is for its robust capabilities, we constantly strive for a better-performing product with more user-oriented features.
For each subproject, we set quality points for items like code, features, and documentation, with a success criterion for each point. When that point is reached, we call the item ready. Candidly, the interim steps are often steep and problem-filled. Given that each measurement reflects only part of the overall project, we regularly correlate all the measurements for a global view.
Here are a few of the yardsticks that we employ:
- Code coverage — With open-source EMMA, we find out the number of code lines, blocks, methods, and classes that are exposed by the unit and functional tests. Part of that information pinpoints the amount of the code tested as a percentage of the total, defining if we've met the quality criteria. We also define which areas of the code are not tested, called coverage holes, and create new tests to fill them.
- Feature coverage — OpenDS delivers features that customers want, that is, customer requirements. Each feature is recorded as an issue in the Issue Tracker, a tool that monitors defects. This data tells us the state the features are in and their status: Ready for Test or Tested.
- Documentation coverage — To ensure that the documentation is reviewed according to the test plan, we adopt a two-phase documentation review process: a technical review of the content followed by a formal QA review. Like the product features, the documentation is divided into categories—books, chapters, and sections—that are recorded in the Issue tracker. Through this coverage, we measure the percentage of the documentation reviewed over time and identify the reviewers and review status.
- Defect rates — This is a traditional measure. The goal is to have no high-priority bugs open at release time. Our Bug Council constantly studies the defects and assesses the risks to customers. We also plot simple graph trends to gauge how well the project is converging.
As a matter of course, all the tests are open to the community. In the works is a task to post the test reports for the functional tests soon after they become available.
A deeper look into the future? Automation is and will remain a key goal as a matter of economics rather than principle. Also, we strongly encourage the community to join us in testing OpenDS, particularly on platforms and scenarios that are difficult to reproduce in house at Sun. Many members are already contributing their time and efforts and we hope to sign up more soon.
Interested? Please join the OpenDS QA mailing list qa@opends.dev.java.net and visit our wiki for details.
Conclusion: People Are the Key
For an open-source project like OpenDS to succeed, there is no substitute for having skilled, capable, and communicative people on board. OpenDS boasts, worldwide, a group of incredibly talented and experienced contributors who are adept in the project and in today's highly demanding and mature market and who have consistently delivered first-rate software. They include developers, test engineers, support engineers, sustaining engineers, technical evangelists, technical writers, and others, all actively involved.
I feel fortunate and proud to be associated with them all and look forward to the continued success of OpenDS as a superior global directory service.
References
- OpenDS
- Sun developer services
We welcome your participation in our community. Please keep your comments civil and on point. You can optionally provide your email address to be notified of replies—your information is not used for any other purpose. By submitting a comment, you agree to these Terms of Use.
|
 Gary Williams, a staff engineer and the QA lead of OpenDS, has almost 20 years experience in QA and testing of software applications. Born and educated in England, Gary holds a Bachelor's degree in Modern History and a Master's degree in Information Technology. He now lives in France with his wife and three children. Gary enjoys renovating an old French Maison du Village, playing football, and following the Barnsley Football Club.
|
 Marina Sum is a staff writer for Sun Developer Network. She has been writing for Sun since 1989, mostly in the technical arena. Marina blogs on Sun's products, technologies, events, publications, and unsung heroes.
|
|
Print-friendly Version
