Assembling a Software Test Team

By Todd Wasserman

A slow or non-functioning software system can cost an organization money, time, and customers. If a financial services company can't provide life insurance quotes fast enough, customers will go elsewhere. If an auto parts supplier can't get orders quickly enough from its customers, it will build fewer of them, biting into profits.

Given the importance of having a robust software system, Carey Schwaber, an analyst with Forrester Research, of Cambridge, Mass., was surprised to find that software testing is often an afterthought. IT shops habitually pay insufficient attention to performance during development and then conduct "perfunctory tuning" before they deploy it, Schwaber wrote in a recent report.

Testing is important because most companies write their own programs or at least customize them to a certain degree. "Most packaged applications are really a platform for business process automation," says Schwaber.

Most companies customize from 25% to 75% of the software they use, according to Thomas Murphy, a research director for Gartner Group, of Stamford, Conn.

"All companies to some extent write their own software," he says.

That's because no two companies are exactly the same. All have different needs. Some may need to integrate a customer relationship management application with a human resources application and a finance system.

While no one tracks the ratio of homegrown software to packaged applications, a September 2006 survey from Forrester found that new software development accounts for about 25% of 2007 software budgets among companies with more than 1,000 employees. The rest goes to software licenses, subscription fees, and maintenance fees.

Rigorous testing is needed
Once that custom software is developed, analysts stress that it should be put through rigorous testing. There are various ways to do this. Some companies employ people who do nothing else but test software. Others assemble teams to both create and test software applications. One constant, however, is that those who write the code should not do most of the testing.

That's not to say a development team should hand in raw code. In most cases, that team performs a "unit test" to make sure the software does what it's supposed to do and then an "integration test" that makes sure it interfaces with other applications.

At that point, the team should send the code on to testers.  Since there is no universal method of testing software, experts offer some of the following suggestions:

• Set up performance requirements What does the organization want the software to do? After answering that question, Schwaber suggests using scenario-based analysis to determine how the software will perform in real-life situations. For instance, a health insurance company might set up a claim adjudication application to perform in the event that a new pharmacy chain is brought onboard.
• Manage expectations Business stakeholders often have unrealistic expectations about how software should perform. Everyone wants the applications to move at light speed, but do the stakeholders understand how much it will cost to make that happen? Directors should also be aware that testing itself can run into the millions. Schwaber says an unnamed $30 billion-plus pharmaceuticals company spent $11 million on in-house testing for its software. Experienced software testers make between $75,000 and $150,000 a year on average, and the software needed to test a few thousand concurrent users costs several hundred thousand dollars.
• Don't test to a date Aside from costs, there are also time considerations involved in testing software. A report from Gartner estimates that testing typically consumes between 10% and 35%  of work on a project. The difference in percentage is based not on "good" versus "bad" software, but on the due date for deployment. "Most testing teams work for exactly the amount of time between when software is delivered to them and when they must deploy it," the report states. Such a situation leads to a phenomenon Gartner calls "good-enough" testing, where a moderate amount of testing is employed. However, most companies would be better off with more time for testing.
• Take into account special variables The production environment for a grocery chain is different than one for a pharmacy. Was the software created to accommodate the special circumstances of each?

As with anything else, common sense also applies. Good-enough testing may work for an application that isn't mission-critical, but other applications should get all the testing they require.

"The higher the risk, that obviously pushes up the need for more testing," Murphy says. "If it's not going to stop my business if something goes wrong, you don't have to be as vigilant."

Todd Wasserman has more than 15 years' experience writing for The New York Times, The Industry Standard and Business 2.0, among other publications. He is currently news editor for Brandweek magazine.