News from RBCS

Recently, we worked on a high-risk, high-visibility system where performance testing ("Let's just make sure it handles the load") was the last item on the agenda. As luck would have it, the system didn't handle the load, and very long days and nights ensued. Delivery was late, several serious disasters were narrowly averted, and large costs were incurred.

It doesn't have to be this way. Nasty end-of-project performance surprises are avoidable. If you've suffered through one or two of these projects and are looking to avoid them in the future, this case study will provide a roadmap.

This case study involves the testing of Internet appliances-simple devices that allow a user to send and receive email, view simple email attachments, and surf the Web. It is not a complex, general purpose computer, but rather a simple box project. On this project, in addition to testing the the appliances themselves, we also needed to test the servers. A key quality risk for the servers was that they might not be able to handle the 25,000 to 40,000 appliances they would have to support. The project was planned to last about six months. Rather than waiting to see what happened during system test, we created a comprehensive performance strategy that included the following steps:

1. Static performance tests of proposed designs

2. Static performance analysis (simulation) of proposed designs

3. Unit performance tests during actual implementation of the system

4. System performance tests of the implemented system

Most organizations skip steps 1 through 3 and begin step 4 just before system installation. That's often the beginning of a sad story (like the one below). In this case, we went through each step. Since we were responsible for system testing, we'll focus on the process from that perspective....

Click here to read the rest of the article.

If you've finished the article and are wondering, "What would have happened without steps 1-3?" let's suppose there had been no up-front work to create a quality design and test the individual units prior to starting step 4, the system performance testing. Alternate realities are always hard, because human beings can't carry out the experiment of bifurcating the time-space continuum to run identical projects side by side with only certain controlled changes. However, you can draw conclusions from similar projects.

We worked on a project immediately prior to this Internet appliance project, one of similar size and complexity, where the architects and developers skipped steps 1 and 2 and did a poor job of step 3, the unit testing. What unit tests they performed were done in non-representative environments and with invalid usage and load profiles. When time pressures loomed, they decided to curtail unit testing and simply deliver the partially-tested software to us for system testing.

The performance tests revealed a series of fundamental design problems. The software consumed far more memory, CPU resource, and disk space than were available on the servers. These problems were found in an "onion-peeling" fashion, layer by layer, while precious project time slipped by. Since it was too late in the project to re-design the faulty software-or so project management thought-the attempted solutions all involved throwing more hardware at the servers. No sooner was one bottleneck resolved, though, than another one was discovered. Eventually the project failed, and the fact that the high-priced servers could not scale to the desired load levels was a major contributor to that failure.

To see more tips and articles by RBCS President, Rex Black, visit the library at www.rbcs-us.com.