[Updated] Reader Question: Why are There so Many Data Points Outside of the Limits?

14

I got a message from a reader of my book Measures of Success: React Less, Lead Better, Improve More.

The questions were:

“I entered data from Press Ganey patient satisfaction responses (12-month running totals) and since the numbers were so close to each other, the PBC showed a bunch of data points outside the upper/lower process limits. 

When I changed the limits to +- 3 sigma, this issue disappeared.  Have you run into this issue before?  Can you help me to understand the difference and its impact on analysis using the PBC?”

Edit: “12-month running totals” means 12-month moving averages, as I later clarified. I'd rather plot each month's actual data and then use a Process Behavior Chart to filter out “noise” in the data. I've made some up some data to create a situation where we have actual monthly data that creates the same moving averages as shown below.

Run Chart of the Moving Average

A simple run chart, which looks like this, is made from the moving-average patient satisfaction data. There is a pronounced downward trend (although the Y-axis scale goes only from 91 to 94, so please don't be misled by that):

A falling moving-average implies that patient satisfaction is getting worse… but is it getting significantly worse or not? This requires a lot of mental gymnastics, which is why I think it's better to just plot each month's data point.

Process Behavior Chart of the Moving Average

Mathematically, we can do this, but I don't think it's a good idea.

His initial Process Behavior Chart for the moving average numbers looked like this:

If you know the PBC methodology, that chart looks troubling, right? This is what concerned the reader.

But, again, maybe we shouldn't be plotting the moving average numbers… he was plotting what was given to him, but I suggested he push back and try to get actual monthly numbers.

Plotting the Actual Data

Since the reader gave me 25 data points of the “moving average” number (25 months of moving averages), I had to create dummy data for 11 months prior… which led to a total of 36 data points.

I created monthly numbers in a way that led to the moving average numbers being the same as what the reader provided. Periods -10 through Period 0 are made up data for the purposes of illustration.

DateMonthly # 12-Month Moving Average
Period -1093
Period -992
Period -894
Period -795
Period -693.8
Period -596.2
Period -496
Period -392
Period -294
Period -192
Period 091
Period194          93.58
Period290.5          93.54
Period391.1          93.47
Period493.3          93.49
Period593.4          93.36
Period693.6          93.24
Period795.9          93.15
Period893.8          92.97
Period992.9          92.79
Period1093.2          92.64
Period1188.7          92.62
Period1289.7          92.51
Period1391.3          92.28
Period1492          92.10
Period1588          92.14
Period1691.3          91.99
Period1793.4          91.99
Period1894.1          92.03
Period1995.2          91.97
Period2096.5          92.18
Period2193.6          92.26
Period2292          92.15
Period2388.7          92.15
Period2490.5          92.23
Period2592.5          92.32

A run chart of the actual monthly data (not the moving average) might look like the chart below. I've had Excel create a 12-month moving average (shown in red) so you can check my work (or the spreadsheet is here).

The moving average line smooths out variation, but it makes it look things were getting worse because the moving average was declining for a while before flattening out… but the actual data might just be fluctuating. I wonder how many managers would react to that moving average “trend” and demand an explanation?

Now, we need a Process Behavior Chart to tell us if this is all routine variation or if there's any signals to be found.

A Process Behavior Chart for the Monthly Data

We can use the first 25 data points as a baseline for calculating the average and the “lower and upper natural process limits” to see the hypothesis or question of “is this a metric that's fluctuating around the average in a predictable way?”

The formulas for the lower and upper limits are

= AVERAGE +/- 3 * MR-bar / 1.128

The MR-bar is the average of the “moving ranges” or the absolute values between each of the consecutive data points, as shown partially below:

It's 3 Sigma, not 2.66 Sigma

The formula for the limits is an approximation of +/- 3 sigma… but we're using the MR-bar and the statistical constant of 1.128 to estimate this instead of using a calculated standard deviation.

The formula for the limits can be simplified as:

= AVERAGE +/- 2.66 * MR-bar

And that sometimes gets confused with being “2.66 sigma” which is not the case. So, to the second part of the question (which was based on his Process Behavior Chart of the moving average):

“When I changed the limits to +- 3 sigma, this issue [with the moving range PBC] disappeared.”

I wrote back to the reader and explained that the limits were already +/- 3 sigma, so no adjustment was necessary (nor would that be appropriate).

Back to the Monthly Process Behavior Chart

So, let's look at the PBC (X Chart and MR Chart) that's based on the baseline of the 25 monthly data points:

This is clearly not a single predictable system over time… using our three main rules for finding signals, there are signals galore:

  • Rule 1: Any single data point outside of the limits
  • Rule 2: Eight consecutive points on the same side of the average
  • Rule 3: 3 out of 3 (or 3 out of 4) points that are closer to the same limit than they are to the average

Let's mark those signals (all Rule 1 signals below the ldower limit):

I don't see any Rule 2 or Rule 3 signals.

We have three months in which the patient satisfaction score was below the lower limit. Other than those three months, it looks like a “predictable system” that's just fluctuating around an average of 93.

We could ask, “Why was patient satisfaction lower those three months?” The organization should find “special causes” for those data points (or it's the same special cause each month, perhaps).

We shouldn't ask for explanations of other individual months that are not signals.

The organization still might want to form a hypothesis around “how can we improve patient satisfaction?” Notice that's a different question than “how can we improve the scores?”

We'd want to improve actual satisfaction (although our measure is, at best, an estimate of that) and not just distort the numbers (for example, by starting a campaign to ask patients to give better scores whether they are deserved or not).

So, in conclusion, the Process Behavior Chart will help us characterize a system as being:

  • Predictable or
  • Not predictable

And we can do so in a somewhat binary way. When we have a predictable system, it's not a good use of time to ask for or look for an explanation to every up and down in the metric. Don't react to noise… DO react to signals and DO work to improve the system… react less, lead better, and improve more.

Again, Plot the Actual Numbers

I think another conclusion is, when in doubt, plot the actual numbers, not the moving average. The reader said that Press-Ganey provided the moving average number because the number of surveys is small.

I'd still rather see the actual numbers. If there's a lot of noise because of the small sample sizes, the chart will adjust for that by showing wider limits. We can use the PBC to distinguish between signal and noise.

The reader pointed out, in an additional exchange, that plotting the moving average means that the effect of any change we make NOW on patient satisfaction will take a while to show in the moving average chart. That's another argument for plotting the actual numbers.

Please post a comment and join the discussion. Subscribe to get notified about posts daily or weekly.

Mark Graban is an internationally-recognized consultant, author, and professional speaker who has worked in healthcare, manufacturing, and startups. His latest book is Measures of Success: React Less, Lead Better, Improve More. He is author of the Shingo Award-winning books Lean Hospitals and Healthcare Kaizen, as well as The Executive Guide to Healthcare Kaizen. He also published the anthology Practicing Lean that benefits the Louise H. Batz Patient Safety Foundation, where Mark is a board member. Mark is also a Senior Advisor to the technology company KaiNexus.

14 Comments
  1. Rob Worth says

    What did they mean by “12-month running totals”?

    1. Mark Graban says

      I’ll have to ask. I hope they weren’t plotting a moving average. If so, I’d advice them to plot the actual data.

      1. Mark Graban says

        Rob – Yes, they are plotting a 12-month moving average…

  2. Santosh says

    Hi Mark..This looks like a moving avrarage chart where intention is to find even a small shift in the mean…For all practical purposes if you see from Hospital perspective is the patient satisfaction score is stuck between 92 to 93. 5..Need to understand the root causes…Is Hospital happy with the scores? What is the cost of poor quality for a one point decrease in patient sat scores..Ideally improvement has to start when the process is under common causes…and predictable within 3 Sigma..

    1. Mark Graban says

      Good questions. If the chart has become “predictable” (meaning there are no signals in the recent data points — it’s fluctuating between +/- 3 sigma around the average), then the hospital can still look to improve the system and, therefore, results.

      I’d still prefer to look at the actual data points. The actual data will be “noisy” but Process Behavior Charts filter out that noise.

  3. Mark Graban says

    Linkedin discussion:

  4. Scott Hindle says

    At the start the question asked is “Can you help me to understand the difference and its impact on analysis using the PBC?” (meaning between control chart “sigma” limits and the generic “statistics class” standard deviation).

    One discussion is here: https://www.spcpress.com/pdf/DJW260.pdf – for the first time reader it isn’t easy, but the distinction in this paper is all about one of the key foundations of the SPC control chart.
    Perhaps a discussion for a future post? Why can we never use the generic, global standard deviation for effective control charting (process behavior chart)… (Is there a risk that people take your book Measures of Success, enthusiastically develop their own PBC chart template in Excel, but screw everything up by using the wrong SD in their calculation of the process limits?)

    1. Mark Graban says

      Thanks, Scott.

      When we say the PBC is +- 3 sigma, the “sigma” is estimated by 3*MR-bar/1.126.

      You’re right that using a calculated standard deviation is a poor estimate of sigma for these purposes. I’ve seen people make that mistake after reading my book and I’ve tried correcting them when I see that.

      That said, using the “wrong” control chart formula is still better than not using control charts at all, I’d say :-)

      I point to Wheeler and his teaching that the Six Sigma control charts (also used a lot in medicine) — the p-chart, u-chart, np-chart, and c-chart are not really valid in the real world.

      I see a p-chart in an article right now about the percentage of patients who wait longer than some certain arbitrary time. A p-chart makes a bad assumption that EVERY single patient has the EXACT SAME chance of waiting a long time. Queuing theory and the real reality would make it easy to show that assumption is a very bad one.

      Therefore, use the XmR / PBC / individuals and MR chart (whatever you want to call it) and the chart is FAR more robust in the real world…

  5. Mark Graban says

    I’ve significantly edited and updated this post, everyone.

    I think that plotting the 12-month moving average was causing a lot of problems.

    I, instead, plotted monthly-data points (made up and extrapolated from the actual moving averages) to re-do the post using analysis of a Process Behavior Chart that’s based on monthly numbers. It tells a different, more accurate, story.

    The “voice of the process” was garbled by the moving average.

  6. Robert kluttz says

    I’m not sure you’d get anything out of a control chart in this case. The odds of going 2 years without significant enough process changes impacting patient sat that would necessitate recalculating your control limits at least once is pretty low. So your control chart/ control limits really aren’t telling you anything that a standard time series plot would. Even if they did, it wouldn’t necessarily be actionable. Multiple box plots stratified by key variables (age group, payor type, gender, specialty, etc) will tell you a lot more in a much shorter time period. I like control charts, but doing them for the sake of doing them won’t inform performance decisions.

    1. Mark Graban says

      Who is “doing them for the sake of doing them?” A control chart helps answer the question of “Has the metric changed significantly, or is it just fluctuating around an average?”

      Many hospitals try MANY countermeasures that don’t have any real effect on patient satisfaction or their metric.

      Could you take a deeper dive with other statistical methods? Of course.

      1. Robert Kluttz says

        Would you want or expect Patient Satisfaction to be in statistical control? I’d guess that it would be preferable to have a continuous improving trend that resulted in a violation of the Nelson Rules than it would be table-topped process performance resulting in the tight control limits that we might want to see in other processes. And conversely, data points falling inside control limits that are artificially wide because of historical high variation in process performance isn’t really a good thing either.

        I’m just not sure what control limits would tell you that a standard time-series plot wouldn’t from an actionable data perspective.

        1. Mark Graban says

          The first thing a Process Behavior Chart does is tell you IF it’s in statistical control. I’d expect it to be in statistical control if the hospital isn’t making major efforts to improve… that, by definition, sounds like a stable system.

          Yes, continual improvement would be preferred and, as you said, the chart would be “out of control” in a good way — it would show evidence of improvement.

          Here’s a run chart showing the % of survey respondents who gave an overall hospital rating of 9 or 10 (this is a very imperfect measure of true patient satisfaction, and I don’t know why they don’t just plot the average or median score instead of this percentage):

          Does that chart imply that:

          1) The number isn’t as high as you’d like or want it to be?
          2) It’s just fluctuating around an average?

          Many hospitals don’t even look at a run chart. They report the LAST number with a COMPARISON (up or down) from the month before. They get caught in the endless cycle of “hooray we got better!” and “Oh no, what happened?” That’s all just a waste of time.

          The run chart view is helpful IF people understand variation. They might draw a dubious linear trend line that suggests the number will continue to get better if they do nothing. They might still react to every up and down.

          I’d still much rather see a control chart with calculated limits:

          The limits and the rules help me make sure I have a better understanding of the cause-and-effect of improvement attempts. If I make an attempt to change the system and the score blips upward a bit, but is still within the limits, I shouldn’t declare victory or that improvement has really occurred. Again, we don’t do charts just to do them… they help us better understand our improvement efforts.

        2. Mark Graban says

          Here is a good testimonial about the use of control chart (XmR Charts):

Leave A Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Win One of Three Signed Copies of Mark's Book "Measures of Success"Enter Now
+ +