Data Without Context Isn’t Very Helpful; Don’t Overreact to Each Up & Down
Some of the best lessons I have ever learned for my career are from Donald Wheeler's brilliant book Understanding Variation: The Key to Managing Chaos. One idea is that data without context is meaningless. The other is the application of statistical process control (SPC) to management decision making.
In one hospital a while back, I saw they had some performance measures posted in the lobby for the public to see. That's a good example of transparency and the attempt (or the thought) should be applauded. Unfortunately, the data likely didn't have any meaning to the general public (or to hospital employees, I would imagine).
The one piece of data (and the chart) that stood out in particular:
So what do those numbers mean to the general public?
What context is missing?
- What is a “quality panel?” Is this like American Idol?
- What's the maximum (best) score? Out of 5 or out of 10?
- What is the trend over the past few years (not just comparing actual to “target”)?
- How does this score compare to other hospitals? Is 3.58 a good score or a bad score?
- This score is indicative of what risks to patients?
I'd also ask, why was the “target” set as 3.59? Did they plan to have less than perfect quality (whatever perfect would be)? Why is the actual so suspiciously close to the target? Are people gaming the numbers? Why do they even need a target? Would the lack of a target hamper people's efforts at delivering the best patient care? How does the target help improve quality? If the target were 4.5, would the actual be 4.49?
Is the hospital really helping or informing anybody through the posting of these numbers?
It would be much more helpful to plot this quality panel in a run chart or a proper SPC chart (as talked about in this post). Having some context about how this score compares to other hospitals would help (although our goal should be perfect quality, not beating benchmarks or targets).
The second primary lesson from Wheeler and SPC is that managers need to avoid overreacting to every little up and down in the data shown in a run chart or an SPC chart. If you're reacting to noise (aka “common cause variation”) and looking for simple answers to the question of “why were laboratory turnaround times longer than average yesterday?” you might be wasting a lot of time.
There was an insightful letter to the editor in the Globe & Mail paper here in Canada where I am visiting this week (see “Put on a happy face” on this page).
Every time there's a precipitous dip in the markets, you use a photo of a guy cradling his head in his hands, as if the world were coming to an end. It's tiresome, and points to a trope in the news industry that you and your readers buy into, that one or two bad days in the market is some kind of monetary natural disaster. It isn't; it's cyclical.
Great point. Not every single day downturn in the stock markets is indicative of any major event or major trend. Sometimes there is noise in the system. Up 200 points, down 200 points. No need to celebrate or get depressed with every up and down in the data. I wonder how many investors use SPC in making investment decisions, to help separate signal from noise?
If the hospital in the first example plotted their monthly quality panel score, they would do well to not overreact to every small blip up and down in that chart.