Recently I came across the Capers Jones article, “A Short History of the Cost Per Defect Metric”, which I hadn’t read in several years. His article focuses primarily on the “Myth Busting” of a long accepted metric and tenet in the software development industry. The myth is the belief that the “cost-per-defect” of a bug found later in the development process is more expensive than when found earlier. While Jones explains that this might be correct mathematically, he states that economically, the calculation is inherently flawed.
Jones makes a compelling argument that I completely agree with, which is the “cost-per-defect” metric is not something that organizations should become fixated on, and here’s why:
- The formula is biased by the fact that more defects are naturally found earlier in the development life cycle and usually when peak project costs are yet to be realized.
- As the project progresses the number of defects identified naturally decreases (i.e. denominator in the calculation), but labor costs often increase due to the need for additional resources as it nears implementation.
- Even if relative variable costs for development, testing, and repairs remain somewhat constant, and assuming some portion of project costs are fixed, the metric will always reveal the average cost to be higher as time goes on due to the diminishing number of defects.
- Projects with fewer defects, or more aptly those having better quality, are penalized by the calculation. Quality projects will very likely shorten project timelines, resulting in reduced costs and improved “time to market.”
- Since all defects are not created equal, the basic metric fails to consider the severity or complexity of a defect and what it will take to resolve.
Instead of putting effort into measuring, analyzing, and talking about what to do about the cost per defect, I’d like to stress the importance of focusing your organization on the “Shift-Left” Model.
Shift Left Testing is the process of moving QA activities earlier in the development process. This could take the form of engaging QA resources and practices as part of Agile teams or by introducing automated testing as part of Continuous Integration and Continuous Deployment. It can be especially effective when assisting DevOps or where speed is required for supporting the Digital Technology domain.
Shift Left Testing is valuable for Waterfall and Iterative projects as well. By being able to conduct static code reviews early, which is an excellent quality practice, or performing functional testing during the development phase, Shift Left exposes issues sooner so they can be addressed in advance of the traditional QA life cycle.
Shift Left should not be done merely to show a decrease in cost-per-defect. There are tangible benefits that will be realized by performing QA activities earlier. Some examples include:
- better quality and customer experience
- quicker project delivery translating into reduced project costs
- improved “time to market” resulting in sooner revenue recognition
…all of which contribute to improving your company’s bottom line!
So, I would strongly recommend utilizing a Shift Left program for your organization to realize the benefits described above.