Return to Bad Software: What To Do When Software Fails.


Quality Cost Analysis: Benefits and Risks

Copyright © Cem Kaner. All rights reserved.

Published in Software QA, Volume 3, #1, 1996, p. 23.


"Because the main language of [corporate management] was money, there emerged the concept of studying quality-related costs as a means of communication between the quality staff departments and the company managers." 1

Joseph Juran, one of the world's leading quality theorists, has been advocating the analysis of quality-related costs since 1951, when he published the first edition of his Quality Control Handbook. Feigenbaum made it one of the core ideas underlying the Total Quality Management movement.2 It is a tremendously powerful tool for product quality, including software quality.

What is Quality Cost Analysis?

Quality costs are the costs associated with preventing, finding, and correcting defective work. These costs are huge, running at 20% - 40% of sales.3 Many of these costs can be significantly reduced or completely avoided. One of the key functions of a Quality Engineer is the reduction of the total cost of quality associated with a product.

Here are six useful definitions, as applied to software products. Figure 1 gives examples of the types of cost. Most of Figure 1's examples are (hopefully) self-explanatory, but I'll provide some additional notes on a few of the costs:4

Figure 1. Examples of Quality Costs Associated with Software Products.

Prevention

Appraisal
  • Staff training 
  • Requirements analysis 
  • Early prototyping 
  • Fault-tolerant design 
  • Defensive programming 
  • Usability analysis 
  • Clear specification 
  • Accurate internal documentation 
  • Evaluation of the reliability of development tools (before buying them) or of other potential components of the product 
  • Design review 
  • Code inspection 
  • Glass box testing 
  • Black box testing 
  • Training testers 
  • Beta testing 
  • Test automation 
  • Usability testing 
  • Pre-release out-of-box testing by customer service staff 

Internal Failure

External Failure
  • Bug fixes 
  • Regression testing 
  • Wasted in-house user time 
  • Wasted tester time 
  • Wasted writer time 
  • Wasted marketer time 
  • Wasted advertisements7 
  • Direct cost of late shipment8 
  • Opportunity cost of late shipment 
  • Technical support calls9 
  • Preparation of support answer books 
  • Investigation of customer complaints 
  • Refunds and recalls 
  • Coding / testing of interim bug fix releases 
  • Shipping of updated product 
  • Added expense of supporting multiple versions of the product in the field 
  • PR work to soften drafts of harsh reviews 
  • Lost sales 
  • Lost customer goodwill 
  • Discounts to resellers to encourage them to keep selling the product 
  • Warranty costs 
  • Liability costs 
  • Government investigations10 
  • Penalties11 
  • All other costs imposed by law 

What Makes this Approach Powerful?

Over the long term, a project (or corporate) cost accounting system that tracks quality-related costs can be a fundamentally important management tool. This is the path that Juran and Feigenbaum will lead you down, and they and their followers have frequently and eloquently explained the path, the system, and the goal.

I generally work with young, consumer-oriented software companies who don't see TQM programs as their top priority, and therefore my approach is more tactical. There is significant benefit in using the language and insights of quality cost analysis, on a project/product by project/product basis, even in a company that has no interest in Total Quality Management or other formal quality management models.12

Here's an example. Suppose that some feature has been designed in a way that you believe will be awkward and annoying for the customer. You raise the issue and the project manager rejects your report as subjective. It's "not a bug." Where do you go if you don't want to drop this issue? One approach is to keep taking it to higher-level managers within product development (or within the company as a whole). But without additional arguments, you'll often keep losing, without making any friends in the process.

Suppose that you change your emphasis instead. Rather than saying that, in your opinion, customers won't be happy, collect some other data:13

You won't get cost estimates from everyone, but you might be able to get ballpark estimates from most, along with one or two carefully considered estimates. This is enough to give you a range to present at the next project meeting, or in a follow-up to your original bug report. Notice the difference in your posture:

Along with arguing about individual bugs, or groups of bugs, this approach opens up opportunities for you (and other non-testers who come to realize the power of your approach) to make business cases on several other types of issues. For example:

Implementation Risks

Gryna (1988)14 and Juran & Gryna (1980)15 point out several problems that have caused cost-of-quality approaches to fail. I'll mention two of the main ones here.

First, it's unwise to try to achieve too much, too fast. For example, don't try to apply a quality cost system to every project until you've applied it successfully to one project. And don't try to measure all of the costs, because you probably can't.16

Second, beware of insisting on controversial costs. Gryna (1988)17 points out several types of costs that other managers might challenge as not being quality-related. If you include these costs in your totals (such as total cost of quality), some readers will believe that you a padding these totals, to achieve a more dramatic effect. Gryna's advice is to not include them. This is usually wise advice, but it can lead you to underestimate your customer's probable dissatisfaction with your product. As we see in the next section, down that road lies LawyerLand.

The Dark Side of Quality Cost Analysis

Quality Cost Analysis looks at the company's costs, not the customer's costs. The manufacturer and seller are definitely not the only people who suffer quality-related costs. The customer suffers quality-related costs too. If a manufacturer sells a bad product, the customer faces significant expenses in dealing with that bad product.

The Ford Pinto litigation provided the most famous example of a quality cost analysis that evaluated company costs without considering customers' costs from the customers' viewpoint. Among the documents produced in these cases was the Grush-Saunby report, which looked at costs associated with fuel tank integrity. The key calculations appeared in Table 3 of the report:18
 

Benefits and Costs Relating to Fuel Leakage 

Associated with the Static Rollover Test Portion of FMVSS 208

Benefits 

Savings - 180 burn deaths, 180 serious burn injuries, 2100 burned vehicles

Unit Cost -- $200,000 per death, $67,000 per injury, $700 per vehicle

Total Benefit - 180 x ($200,000) + 180 x ($67,000) + 2100 x ($700) = $49.5 million

Costs 

Sales - 11 million cars, 1.5 million light trucks.

Unit Cost -- $11 per car, $11 per truck

Total Cost - 11,000,000 x ($11) + 1,500,000 x ($11) = $137 million

In other words, it looked cheaper to pay an average of $200,000 per death in lawsuit costs than to pay $11 per car to prevent fuel tank explosions. Ultimately, the lawsuit losses were much higher.19

This kind of analysis didn't go away with the Pinto. For example, in the more recent case of General Motors Corp. v. Johnston (1992)20, a PROM controlled the fuel injector in a pickup truck. The truck stalled because of a defect in the PROM and in the ensuing accident, Johnston's seven-year old grandchild was killed. The Alabama Supreme Court justified an award of $7.5 million in punitive damages against GM by noting that GM "saved approximately $42,000,000 by not having a recall or otherwise notifying its purchasers of the problem related to the PROM."

Most software failures don't lead to deaths. Most software projects involve conscious tradeoffs among several factors, including cost, time to completion, richness of the feature set, and reliability. There is nothing wrong with doing this type of business tradeoff, consciously and explicitly, unless you fail to take into account the fact that some of the problems that you leave in the product might cost your customers much, much more than they cost your company. Figure 2 lists some of the external failure costs that are borne by customers, rather than by the company.

Figure 2. Comparison of External Failure Costs Borne by the Buyer and the Seller

 

Seller: external failure costs

Customer: failure costs
These are the types of costs absorbed by the seller that releases a defective product.  These are the types of costs absorbed by the customer who buys a defective product. 
  • Technical support calls 
  • Preparation of support answer books 
  • Investigation of customer complaints 
  • Refunds and recalls 
  • Coding / testing of interim bug fix releases 
  • Shipping of updated product 
  • Added expense of supporting multiple versions of the product in the field 
  • PR work to soften drafts of harsh reviews 
  • Lost sales 
  • Lost customer goodwill 
  • Discounts to resellers to encourage them to keep selling the product 
  • Warranty costs 
  • Liability costs 
  • Government investigations 
  • Penalties 
  • All other costs imposed by law 
  • Wasted time 
  • Lost data 
  • Lost business 
  • Embarrassment 
  • Frustrated employees quit 
  • Demos or presentations to potential customers fail because of the software 
  • Failure when attempting other tasks that can only be done once 
  • Cost of replacing product 
  • Cost of reconfiguring the system 
  • Cost of recovery software 
  • Cost of tech support 
  • Injury / death 

The point of quality-related litigation is to transfer some of the costs borne by a cheated or injured customer back to the maker or seller of the defective product. The well-publicized cases are for disastrous personal injuries, but there are plenty of cases against computer companies and software companies for breach of contract, breach of warranty, fraud, etc.

The problem of cost-of-quality analysis is that it sets us up to underestimate our litigation and customer dissatisfaction risks. We think, when we have estimated the total cost of quality associated with a project, that we have done a fairly complete analysis. But if we don't take customers' external failure costs into account at some point, we can be surprised by huge increased costs (lawsuits) over decisions that we thought, in our incomplete analyses, were safe and reasonable.


1 Gryna, F. M. (1988) "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, page 4.2.

2 Feigenbaum, A.V. (1991, 3rd Ed. Revised), Total Quality Control, McGraw-Hill, Chapter 7.

3 Gryna, F. M. "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, page 4.2. I'm not aware of reliable data on quality costs in software.

4 These are my translations of the ideas for a software development audience. More general, and more complete, definitions are available in Campanella, J. (Ed.) (1990), Principles of Quality Costs, ASQC Quality Press, as well as in Juran's and Feigenbaum's works.

5 Principles of Quality Costs, ASQC Quality Press, Appendix B, "Detailed Description of Quality Cost Elements."

6 "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.

7 The product is scheduled for release on July 1, so your company arranges (far in advance) for an advertising campaign starting July 10. The product has too many bugs to ship, and is delayed until December. All that advertising money was wasted.

8 If the product had to be shipped late because of bugs that had to be fixed, the direct cost of late shipment includes the lost sales, whereas the opportunity cost of the late shipment includes the costs of delaying other projects while everyone finished this one.

9 Note, by the way, that you can reduce external failure costs without improving product quality. To reduce post-sale support costs without increasing customer satisfaction, charge people for support. Switch from a toll-free support line to a toll line, cut your support staff size and you can leave callers on hold for a long time at their expense. This discourages them from calling back. Because these cost reductions don't increase customer satisfaction, the seller's cost of quality is going down, but the customer's is not.

10 This is the cost of cooperating with a government investigation. Even if your company isn't charged or penalized, you spend money on lawyers, etc.

11 Some penalties are written into the contract between the software developer and the purchaser, and the developer pays them if the product is late or has specified problems. Other penalties are imposed by law. For example, the developer/publisher of a computer program that prepares United States taxes is liable for penalties to the Internal Revenue Service for errors in tax returns that are caused by bugs or design errors in the program. The publishers are treated like other tax preparers (accountants, tax lawyers, etc.). See Revenue Ruling 85-189 in Cumulative Bulletin, 1985-2, page 341.

12 I am most definitely not saying that a tactical approach is more practical than an integrated, long-term approach. Gryna notes that there are two common approaches to cost-of-quality programs. One approach involves one-shot studies that help the company identify targets for significant improvement. The other approach incorporates quality cost control into the structure of the business. (Gryna, 1988, in Juran, J. M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.2 onward.) The one-shot, tactical approach can prove the benefit of the more strategic, long-term system to a skeptical company.

13 Be sensitive to how you do this. If you adopt a tone that says that you think the project manager and the programming staff are idiots, you won't enjoy the long-term results.

14 in Juran, J. M. & Gryna, F. M. (1988, 4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.27-4.28.

15 Quality Planning and Analysis (2nd Ed.), McGraw-Hill,pages 30-32. Also, see Brown, M. G., Hitchcock, D. E. & Willard, M. L. (1994), Why TQM Fails and What To Do About It, Irwin Professional Publishing.

16 As he says in Juran, J.M. (1992), Juran on Quality by Design, The Free Press, p. 119, Costs of poor quality "are huge, but the amounts are not known with precision. In most companies the accounting system provides only a minority of the information needed to quantify this cost of poor quality. It takes a great deal of time and effort to extend the accounting system so as to provide full coverage. Most companies have concluded that such an effort is not cost effective. [¶] What can be done is to fill the gap by estimates, which provide managers with approximate information as to the total cost of poor quality and as to where the major areas of concentration are."

17 "Quality Costs" in Juran, J.M. & Gryna, F. M. (1988,4th Ed.), Juran's Quality Control Handbook, McGraw-Hill, pages 4.9 - 4.12.

18 This table is published in Keeton, W. P., Owen, D.G., Montgomery, J. E., & Green, M.D. (1989, 2nd Ed.) Products Liability and Safety, Cases and Materials, Foundation Press, page 841 and Posner, R.A. (1982) Tort Law: Cases and Economic Analysis, Little Brown & Co., page 225.

19 Grimshaw v. Ford Motor Co. (1981), (California Court of Appeal), California Reporter, volume 174, page 348.

20 Southern Reporter, 2nd Series, volume 592, pages 1054 and 1061.



 

Cem Kaner consults on technical and management issues, practices law, and teaches within the software development community. His book, Testing Computer Software, received the Award of Excellence in the Society for Technical Communication's 1993 Northern California Technical Publications Competition. He has managed every aspect of software development, including software development projects, software testing groups and user documentation groups. He has also worked as a programmer, a human factors analyst / UI designer, a salesperson, a technical writer, an associate in an organization development consulting firm, and as an attorney (typically representing customers and software development service providers). He has also served pro bono as a Deputy District Attorney, as an investigator/mediator for Santa Clara County's Consumer Affairs Department, as an Examiner for the California Quality Awards. He holds a B.A. (Math, Philosophy, 1974), a J.D. (1993), and a Ph.D. (Experimental Psychology, 1984) and is Certified in Quality Engineering by the American Society for Quality Control. He teaches at UC Berkeley Extension, and by private arrangement, on software testing and on the law of software quality. 


Return to Bad Software: What To Do When Software Fails.

The articles at this web site are not legal advice. They do not establish a lawyer/client relationship between me and you. I took care to ensure that they were well researched at the time that I wrote them, but the law changes quickly. By the time you read this material, it may be out of date. Also, the laws of the different States are not the same. These discussions might not apply to your circumstances. Please do not take legal action on the basis of what you read here, without consulting your own attorney.
Questions or problems regarding this web site should be directed to Cem Kaner, kaner@kaner.com.
Last modified: Sunday October 26, 1997. Copyright © 1997, Cem Kaner. All rights reserved.