In recent years, there’s been a movement away from having first-year residents work such long grueling hours – 24 hour shifts or longer, multiple days in a row on call, etc. The idea was that fatigued residents were making poor decisions and mistakes, leading to patient harm. It seems reasonable, since who performs really well when tired? And, that includes nurses at the end of a 12-hour shift.
A recent study suggests that error rates have gone UP with the shorter resident shifts… can we really conclude that from the available data?
As summarized on the TIME blog: Fewer Hours for Doctors in Training Leading to More Mistakes.
First off, we have to be careful about confusing correlation and causation. Does fewer hours LEAD TO more mistakes? Some argue it does, since the residents are receiving fewer hours of training, there are more handoffs, and hospitals aren’t adding more residents to pick up the gap left by shorter shifts.
From the article:
In 2011, those hours were cut even further, but the latest data, published online in JAMA Internal Medicine, found that interns working under the new rules are reporting more mistakes, not enough sleep and symptoms of depression. In the study that involved 2,300 doctors from more than a dozen national hospitals, the researchers compared a population of interns serving before the 2011 work-hour limit was implemented, with interns working after the new rule, during a three-month period. Those in the former group were on call every fourth night, for a maximum of 30 hours, while the latter group worked no more than 16 hours during any one shift. They gathered self-reported data from on their duty hours, sleep hours, symptoms of depression, well-being and medical errors at three, six, nine and 12 months into their first year of residency.
”In the year before the new duty-hour rules took effect, 19.9% of the interns reported committing an error that harmed a patient, but this percentage went up to 23.3% after the new rules went into effect,” said study author Dr. Srijan Sen, a University of Michigan psychiatrist in a statement. “That’s a 15% to 20% increase in errors â€” a pretty dramatic uptick, especially when you consider that part of the reason these work-hour rules were put into place was to reduce errors.”
Let’s reflect on the scale of the numbers for a moment — roughly 20% of interns REPORT committing an error that harmed a patient. How many interns did NOT report such harm, given healthcare’s problem with the underreporting of errors? There’s bound to be a tendency to underreport even in an anonymous survey, as pride and one’s unwillingness to admit errors coming into play.
The study, as broad as it was, was a simple before and after study. That means TWO data points. Two data points do not make a trend (as I’ve written about before, in the context of consulting firm before and after studies).
Here is the comparison in a simple two-point chart:
If you’re somebody who was pre-disposed AGAINST the shortened work hours, this data (the two data points) would certainly seem to “prove” your case.
All we know is that the second data point is higher. We don’t have a longer time series of data that would allow us to evaluate a trend or make a more statistically meaningful conclusion through the use of a Statistical Process Control chart (see a webinar I did on this topic for Gemba Academy).
If we viewed each data point like a political poll, they would report a margin of error, such as +/- 2%. With the above patient harm data, if the “Before” number was understated and the “After” number was overstated, then it’s POSSIBLE that the level of patient harm is flat or that it’s even decreased a touch with the shorter intern hours.
I make a similar point in my SPC talks about simple year-on-year before/after data from my time at Dell Computer:
Looking at a reasonable margin of error (based on monthly variation, which I had access to) in this normalized “Ship to Target” number, it’s possible “the number” went up or it went down. The annual number of 104 for the year 2000 wasn’t definitively higher than the 100 value for the year 1999. Yes, 104 is higher than 100, but not necessarily, when you consider the variation in the data. Showing two data points often simplifies things in a number of ways. The same idea applies to the patient harm data.
It could be that the “pretty dramatic uptick” (as it was called) in the data is really just part of noise in a fairly stable system. If we had more studies looking at 3-month periods over time, we might see a chart like the one below (with TOTALLY made up data, other than those two 19.9% and 23.3% data points, the last two in the chart):
That jump between the last two data points (from 19.9% to 23.3%) might be “pretty dramatic” but it might also not be unusual. It could be “common cause” variation in a system. Without more data, we don’t know. I’d be VERY curious to see the next data point, if this was studied for another three months. If the 23.3% data point were part of the noise in the system, the next data point could be just as likely 18%…
People seem to be making a “special cause” assumption – shortening the hours resulted in higher reported error rates, or at least the headline assumes that.
With different made up and assumed data, the chart could look this… which might imply a special cause data point that’s at least a bit out of line with the history:
There COULD be higher reported error rates… but let’s also keep in mind that hospitals, medical schools, and the broader healthcare community have been placing much more emphasis on not covering up errors or problems — more honesty and transparency and less “shame and blame” punishment.
Higher REPORTED rates of harm doesn’t mean more ACTUAL HARM is occurring.
So we do we know from this study? We know that shorter working hours doesn’t seem to be a panacea, as the interns aren’t reporting benefit:
Although the trainees working under the current work rules spent fewer hours at the hospital, they were not sleeping more on average than residents did prior to the rule change, and their risk of depression remained the same, at 20%, as it was among the doctors working prior to 2011.
Another possible cause of the possible increase in reported harm – handoffs.
Another source of errors came as one intern going off duty handed his cases to another. With fewer work hours, the researchers say the number of handoffs has increased, from an average of three during a single shift to as many as nine. Anytime a doctor passes on care of a patient to another physician, there is a chance for error in communicating potential complications, allergies, or other aspects of the patient’s health; that risk is boosted when the transition occurs several times over.
If handoffs are error prone, then I’d say we need to FIX handoff communication process. We shouldn’t have a knee-jerk reaction to doing fewer handoffs. We can do them better, with Lean and Six Sigma process improvement methodologies.
I guess I’d agree with the conclusion of the piece (which is far less definitive than the headline):
Figuring out the right balance between humane work conditions that promote the best learning environment for residents and the highest quality of care for patients may still be a work in progress. More research is needed to pinpoint what’s driving the uptick in medical errors and determining the best strategies for improving resident training to bring these rates down.
And I’d add, “Be careful about drawing strong conclusions from just two data points!”
Also worth reading are the comments to the post, including the stereotypical doctor response (generations have been trained this way, I did it, and this generation is soft) and allegations of lying about the number of hours worked to keep under the regulation limits.
I’ll be blogging tomorrow about other examples of people fudging data to make things look good – in education and in healthcare.
About LeanBlog.org: Mark Graban is a consultant, author, and speaker in the “lean healthcare” methodology. Mark is author of the Shingo Award-winning books Lean Hospitals and Healthcare Kaizen, as well as the new Executive Guide to Healthcare Kaizen. Mark is also the Chief Improvement Officer for the technology company KaiNexus.