January 1, 2013

Is Intrinsic Quality Important?

These observations are derived from quality benchmark studies carried out by the author and his colleagues between 1984 and 2009 in several hundred companies in the United States, Europe, and Japan.

  • Leading edge enterprises employ technologies that can approach 99% cumulative defect removal rates.
  • The norm for US firms is a cumulative defect removal rate of 75%.
  • A cumulative defect removal rate of 95% on a project appears to be a nodal point where several other benefits accrue. For projects of similar size and type, these projects:
      • have the shortest schedules.
      • have the lowest quantity of effort in terms of person-months
      • have the highest levels of user satisfaction after release.

Reaching the Nodal Point of Intrinsic Quality

  • Companies that depend purely upon testing for defect removal almost never top 90% in cumulative defect removal, and often are below 75%.
  • The defect removal efficiency of TDD is higher than many forms of testing and can top 85%.
  • However, even with TDD a phenomenon called “bad-fix injection” needs to be factored in to the equation. About 7% of attempts to fix bugs accidentally include new bugs in the fixes themselves.
  • If TDD is combined with other approaches such as formal inspection of the test cases and static analysis of the code then defect removal efficiency can top 95%.