Selecting Appropriate Metrics

Case 1: The problem of A&B outlined above expresses the problem using a simple example that will explain to all -- Six Sigma practitioners and operating personnel alike -- the issues involved and the methods used for their solution.

Case 2: A real-life case of selecting appropriate metrics encountered while introducing Six Sigma for a large company in the business of digitizing data is more complex. In this case the company had selected the theme of "Improving Quality Consistency" in its operations. When the work was begun the metric being used to measure quality was number of character errors expressed as a % of the total number of characters output in a day.

Step 1: Why is the measurement required?
Or more rephrase the question more precisely -- why is a new metric required?

Case 1: To reduce the time wasted due to parties A&B not arriving at meetings at the same time.

Case 2: When asked to describe the current state of "Quality Consistency," it was found that the organization was measuring the percentage of correct characters in sampled batches and quality was expressed as the average % of characters that were correct. This average was varying at different times. Everyone knew it was varying but no one knew how to measure variability. Thus, one was given figures like 99.95%, 99.952%, 99.992%, etc., for the daily averages of error percentages. The metric was not measuring the problem -- inconsistency. Nor was it measuring process outputs. Instead, it was measuring snapshot average quality which led to sporadic knee-jerk problem solving whenever the readings were out of range.

Step 2: What needs to be measured?
Case 1: The obvious answer is that time needs to be measured more precisely so that waste can be avoided.

Case 2: Firstly, the problem relates to errors and therefore rather than measuring the correct characters the percentage of error characters should be measured. Thus, instead of stating quality as 99.92% correct it would be expressed much better as 0.05%, .048% .008% defectives for the purpose of error control. These figures are a bit harder to handle than the figures given above but this aspect will be dealt with in Step 3.

Secondly and more importantly, the standard deviation of errors needs to be introduced as a measure of consistency of quality. This will directly measure the customer problem of inconsistent quality and therefore focus efforts to improve it.

Step 3: What is the precision of measurement required?
This may seem a trivial step but is actually very important. Engineers will recognize this as specifying the least count of a metric.

Case 1: To improve the accuracy of the measurement, a metric measuring smaller intervals of the variable time is required. How small? Suppose A&B decides to divide the day into 24 smaller metrics -- i.e. select one hour as the unit. A&B will then agree to come at an appointed hour the next day. However, one may come at say 11 am while the other might come any time between 11 am and 12 am and still claim the time is 11 am. The wait (of up to 59 minutes) would in many cases be still intolerably long.

So A&B would conclude they need a more precise metric. They could then decide to select a metric, which was 1/3600th of an hour, i.e. a second. In this case, the specification for a designated time might be 2100 seconds after 11 am. They would quickly realize that this unit was too small and waiting for say 300 seconds was not such a big deal. So perhaps they would come to 60 parts of the hour -- a minute -- as the appropriate metric.

It would be easy to measure, convenient to deal with, and easy to interpret. It should be appreciated that a year, century, month, week and millisecond are all time metrics and can be used to specify time -- but try to specify the time of a meeting using any of these and you will quickly conclude that they are not appropriate for A&B's requirement.

Case 2: When queried about the customer specifications, the reply was that the customer accepts a minimum accuracy in data of 99.995%. So a problem was felt whenever sample quality checks indicated batches/lots exceeding this standard. Typical figures in quality reports for successive monthly averages would read as follows:

99.998%    OK
99.992%    Failed
99.997%    OK
99.991%    Failed

If a staff member reported that the average quality had improved from 99.993% in the previous month to 99.996% in the next month, it was not possible to estimate the percentage improvement. Following step 2, when it was realized that measurement of average % of errors was more suitable for this application, the readings changed to:

.002% character errors
.008% character errors
.003% character errors
.009% character errors

The customer standard now became less than .005% errors permissible. Suddenly one could see that the average errors were 400% more in month 2 compared to month 1.

These decimals were inconvenient to handle, however. For example, a 50% improvement on .003% meant going to another decimal place -- .0015% -- and so on. It was then realized that if the errors were calculated in per million (ppm defectives), instead of percent, the readings would become much easier to handle as shown below:

20 ppm
80 ppm
30 ppm
90 ppm

And the customer standard on this scale became less than 50 ppm errors permissible. This became easy to handle, express and grasp. A far greater precision of measurement -- from % to per million -- was warranted. The final metric selected was ppm errors instead of % correct.

Step 4: How will it be measured?
Case 1: It is apparent that as A&B went from days to hours and minutes they had to develop measuring instruments that progressed from sundials to mechanical and then quartz watches. As atomic physicists worked with problems involving the need to measure precisely -- they developed seconds, milliseconds and nanoseconds, and atomic clocks were required to measure them.

Case 2: The customer problem was "inconsistent quality" and the average of errors measured was not measuring this variable. So how was variability to be measured? The obvious answer is through standard deviation, but as is so often the case it was not obvious to the operating personnel. After training them in the concept and the calculation of sigma, this metric was introduced. As the first sigma readings came out and were interpreted a number of very important changes begun to take place.

The concepts of variability and controlling variability to satisfy the customer became clear. So the whole understanding of quality changed as manifested in the following statements before and after the introduction of the sigma metric:

Before: "The customer quality limit is < 50 ppm errors"

After: "We have to produce so that the (average + 3*sigma) of errors is < 50 ppm."

Suddenly it dawned on everyone that the average would have to be much less than 50 ppm to ensure that the customer was satisfied despite the variability. And that even with this standard they would be out of specifications .3% of the time. Gradually the whole effort shifted to reducing sigma, i.e. variability. The results were dramatic and are described in the Six Sigma case study: Converting Paper to Electronic Documents.

In fact, the introduction of the sigma metric created a mindset change and all the customer lines were shifted to average and sigma control charts for both error and turnaround measurements. The latter change can be understood by referring to Applying Lean Manufacturing To Six Sigma - A Case Study.

Step 5: What use will the measurement be put to? By whom?
"What is not measured cannot be improved," describes the tremendous emphasis that Six Sigma places on measurement. Yet it also cautions that measurements that are not used should be avoided. One of the reasons that a measurement might go unused is if the metric is inappropriate for the user. Among other reasons, this may be because it is not understood, does not serve a purpose, or is too cumbersome or complex to use.

Case 1: If the metric selected to measure time was seconds or years then A&B would not have adopted the metric for all the reasons listed above. The metric of minutes helped them and so they would adopt it readily.

Case 2: Both ppm and sigma metrics were readily adopted once the concept, calculation and utility were clearly understood by the personnel as it helped them in improving and controlling their key problem. In the case of ppm this was easy, but in the case of sigma a very simple explanation of the concept had to be developed and explained repeatedly to a number of personnel at all levels until it was internalized and its interpretation as familiar as that of "average." Today it has become a way of life and most personnel wonder how they were working without it.

Conclusion
These examples are intended to help the inexperienced Six Sigma practitioner recognize the issues involved and the criticality of ensuring that the metrics selected and used at all levels of the organization are the appropriate ones. Fashioning appropriate metrics requires knowledge, experience and a lot of common sense. These examples can be used to explain to key operations personnel the concepts of metrics and why they might need to create new metrics or adjust current metrics.

About The Author
Niraj Goyal has 25 years of rich and varied working experience in multinationals in various operating roles, among them Operations Director, Cadbury India Limited, where he was exposed to and was among the leading implementers of the TQM movement. A few years back he set up his own consultancy, Cynergy Creators Private Limited. Mr. Goyal consults in India and US with a diversity of industries - training them and facilitating the implementation of the techniques of TQM and Six Sigma until the culture of continuous change is internalized. Further details about this article and assistance for similar change initiation and implementation through training and facilitation can be obtained from Mr. Goyal at nirajgoyal@vsnl.in.