Six Sigma Software Metrics, Part 1

	Bookmark This Page Email This Page Format for Printing Cite This Article Submit an Article Read More Articles
	Related Tools & Articles Six Sigma for Software - A Role Based System Six Sigma for Software… More Than a New Tool Software Quality Function Deployment
	Discussion Forum "How does one quantify the defects in software for a six sigma initiative? Is it reasonable to predict the probability of residual defects based on defects detected per test cycle?" Six Sigma Metrics For Software

Download Products Six Sigma Green Belt Training Slides - 2009 Version Calibration System Software - Calibro CSSBB Preparation Pack - 2009 Version

By David L. Hallowell

Six Sigma brings sharp focus to customer and business requirements and the defects connected with the failure to satisfy them. While the relevance of that view is clear enough to software professionals, their introduction to Six Sigma is often gets stopped short in questions about how the notions of yield, sigma level, or defects per million opportunities (DPMO) fit their world. As those are the toughest concepts to map to software, this article will suggest it's better to end the Six Sigma metrics discussion there, when a foundation in other defect metrics provides suitable measurement systems and perspective.

This is the first in a series of three articles that views the range of software measures that support Six Sigma software goals and results. Table 1 illustrates how those goals "start simple" with the fundamental need to reduce released defects, progressing to more sophisticated goals where each build on prior work and understanding. Defect density metrics like DPMO appear at the end of the table as a culminating goal for software metrics. Rather than starting by wrestling with questions like "What's an opportunity for a defect?", many software businesses would be better off to work through those goals in sequence, recovering significant dollars with DMAIC projects as they go. By the time the last few goals are tackled, the measurement system and understanding about defects frames the DPMO question in a way that is much more understandable.

"Maturity" In The Use Of Defect Metrics
When Six Sigma originated at Motorola, the sharp focus on defects created an important shift from the common manufacturing "yield" metric¹, which tracks defective units to the more detailed view of defect counting within units. For software, this part of Six Sigma is easy - no shift at all - as defects within units (bugs) have always been a natural measure of quality. While all software organizations find and fix bugs, there is huge variation in the quality of data gathering, measurement (converting the raw data into measures), and use of that data.

Table 1 shows that the progression in defect metrics refinement and richness of data-use can be seen as a series of goals, each of which call for some disciplined processes and which enable particular uses of the data. We will see that there is a lot to do before the question about DPMO even begins to make sense. Further, we'll recognize that a software organization can be fully successful with Six Sigma without even implementing DPMO metrics.

Goal 1: Reduce Released Defects
This fundamental Six Sigma goal is one that all software work deals with in one way or another. Tracking Defect Containment requires, at a minimum, a test process and defect tally system. When combined with post-released defect tallies this basic data supports the primary defect containment metric: Total Containment Effectiveness (TCE). The computation is simple.

Where:
Errors = Potential defects that are found during the phase that created them.
Defects = Errors that escape to a subsequent development or delivery phase.

Obviously, a high TCE is best. While TCE data can be collected and the metric can be easily computed, many software organizations cannot document their performance on this basic metric. Benchmarking shows a range of TCE in the industry from about 65% to 98%, with many organizations somewhere in the 75% - 85% range. In some software businesses where there are particularly strong penalties for escaped defects (high end data storage systems, for example) the TCE can approach and reach 100%.

Figure 1: Total Containment Effectiveness

If the same defects that are counted to compute TCE are classified and grouped according to type and, even better, with respect to where they were inserted into the process, some rudimentary Defect Causal Analysis can be done.

Figure 2: Origins of Defects Found During Test

In Figure 2 we see that, for defects found in test, most were inserted during coding. Building a measurement system for classifying these defects by type would be the next step in uncovering the key defect root causes. That, of course, would help us to reduce or remove root causes. In concert with root cause (defect insertion) reduction we may want to step up efforts to reduce the escape of defects from upstream phases to test. That highlights the next goal.

Goal 2: Find And Fix Defects Closer To Their Origin
This requires that, in concert with the creation of important work products (not just code, of course, but documents like requirements, specifications, user documentation), we integrate activities that are effective in finding the errors that could become downstream defects. Formal work-product inspections, based on Fagan's seminal work at IBM³, have been shown to be the most efficient and effective way to find potential defects and develop useful data about them.

With inspections and perhaps supplementary defect-finding activity in place we could track defect insertion rates at each key development or delivery stage. Figure 3 shows where defects attributed to Design were found. Note that 35% were found "in phase" (during Design) and lesser proportions found by Test, during Coding, and by the Customer (post release).

Figure 3: Pareto Chart For Design Defects

Data like this enables the computation of two related containment metrics: Phase Containment Effectiveness (PCE) and Defect Containment Effectiveness (DCE).

PCE tracks the ability of each phase to find defects before they escape that phase. DCE tracks the ability of each phase to find defects passed to it by upstream phases. Each of these measures provides insight into the strengths and weaknesses of the error and defect detection processes in each phase. Tracking these numbers together with defect type classifications, can identify patterns that shed light on the causes of the types of defects that are found. As a system like this matures, with data over time and across multiple projects, an understanding of the defect "time dynamics" can begin to come into view, setting the stage for the next goal.

The use of a defect containment scorecard to track TCE, DCE, PCE, and Defect Repair Costs is described in this related article: Is Software Inspection Value Added?

Table 1: Software Organization Goals Versus Processes And Metrics  Six Sigma Goal  Required Processes  Enabled Processes  Metrics 1. Reduce Released Defects Unit/Integration/System Test Pre-release vs. Post-release Defect Tallies Total Containment Effectiveness (TCE) Focus defect fix/removal work Operational definitions for classifying Defects Problem-solving knowledge-base Basic causal analysis Defect stratification by type Poisson Modeling for defect clustering   (Optional) Define the "Unit" for work-products delivered Yield assessments for work-product units delivered Yield predictions for work-product units planned Total Defects per Unit (TDU) Rolled Throughput Yield (RTY) 2. Find and Fix Defects Closer to Their Origin Upstream Defect Detection Process (Inspections) Defect Insertion rates and Defect Find Rates for phases or other segments of work breakdown structure (WBS) Phase Containment Effectiveness (PCE) and Defect Containment Effectiveness (DCE) Gather data necessary for process monitoring and improvement Defect Sourcing Process Improved causal analysis Defects per Unit (DPU) for phases or other segments of WBS, contributions to TDU 3. Predict Defect Find and Fix Rates During Development and After Release Rayleigh or other time-dynamic modeling of defect insertion and repair Defect Estimating Model - calibrated to the history and current state in our process Given data on defects found during upstream development or WBS stages, predict defect find rates downstream. Predicted total defects for an upcoming project of specified size and complexity Best fit Rayleigh Model Predictive Rayleigh Model 4. Compare Implementations Within the Company Company's choice of appropriate normalizing factors(LOC, FP, etc) to convert defect counts into meaningful defect densities Defect Density comparing groups, sites, code-bases, etc. within the company Defects / Function Point Defects per KLOC 5. Benchmark Implementations Across Companies Define Opportunities for counting defects in a way that is consistent within the company and any companies being benchmarked Defect Density comparing performance with other companies (and, if applicable, other industries) DPMO Sigma Level Cpk z-score

Looking Ahead
In part 2 we'll look at Goal 3: Predict Defect Find and Fix Rates During Development and After Release. Building on the foundation provided by containment measures, we will explore defect arrival rate prediction models and tests for their adequacy. In part 3 we'll cover Goals 5 and 6 related to defect density measures, including the elusive Defects per Million Opportunities (DPMO).

Read Six Sigma Software Metrics, Part 2 »

About The Author
David L. Hallowell is a Managing Partner of Six Sigma Advantage, Inc. and has 20+ years experience as an engineer, manager and Master Black Belt. As Digital’s representative to Motorola’s Six Sigma Research Institute he worked on the original courseware for Black Belts and the application of Six Sigma to software. Mr. Hallowell has supported Six Sigma deployments and trained many Black Belts and trainers worldwide.

With a special focus on Design for Six Sigma, he has led development teams to breakthrough improvements in the concept development and design of a number of commercial products. Mr. Hallowell has patents and publications in the area of microelectronics packaging and high speed interconnect. He has authored courses in Software DFSS, Design of Experiments, C++ , and Computational Intelligence Tools. He is a co-founder of Six Sigma Advantage Inc., where he co-authored the Black Belt, Green Belt, and Foundation curriculum. He can be reached via email at dhallowell@6siga.com.

Footnotes And References
1 Yield = good units / units started. "First Time Yield" (FTY) computes this quantity before any rework.
2 FP = Function Points, measure software work-product size based on functionality and complexity. While this system is independent of implementation language, there are conversion factors that map function point counts to 'lines of code' in cases where code is the work-product being sized. As order of magnitude rule of thumb 1 FP converts to about 100 lines of C code.
3 M.E.Fagan, "Design and Code Inspections to Reduce Errors in Program Development," IBM Systems Journal, vol. 15, No. 3, 1976.

BEST SELLING PRODUCTS (iSixSigma Publications) Six Sigma Black Belt (DMAIC) Training Slides - 2009 Version! The 2009 Six Sigma Black Belt course includes over 40 more slides than the 2008 version. Contents include: 1,220 PowerPo... Certified Lean Six Sigma Black Belt Assessment Exam Interested in assessing your knowledge of Lean Six Sigma? Preparing for certifications? Testing your students and traine... Certified Lean Six Sigma Green Belt Assessment Exam This assessment exam is useful for students interested in assessing their knowledge of Lean Six Sigma on the Green Belt ... Kaizen Workshop E-book This 150+ page ebook teaches key tools and techniques of Kaizen, as well as real application to enhance learning. Kaizen... Certified Lean Six Sigma Black Belt E-book In 670 pages learn everything within the Lean Six Sigma DMAIC body of knowledge to successfully achieve Black Belt certi... Process Management Training Slides The 2008 Process Management course is designed in two phases comprised of:352 Powerpoint slidesInstructor notesSlide exp... Design For Six Sigma (DFSS) E-Book or Print Need an "encyclopedia" consisting of many of the tools you’ll study? Need a helpful refresher to apply the DFSS process?... Sell Your Products