Lies, Damn Lies, and Vanity Metrics – in Policing and Healthcare
I arrived in San Francisco yesterday for the Lean Startup conference, where I gave my 5-minute Ignite format talk on the parallels between Lean hospital design and Lean startups – Lean is Lean. There will be video available of the talk online and I'll share it when I can. Follow the #LeanStartup hashtag or @LeanStartup on twitter for the latest updates from the event.
When I got to my hotel, I saw the front page headline on the San Francisco Chronicle: “Arrest totals down sharply in Oakland.” The article says arrests are down 44%. That sounds great… oh, but wait… is crime actually down 44%??
Nope… from the article:
The arrest figures, obtained under the state's Public Records Act, raise questions about the effectiveness and assertiveness of the Police Department, which is struggling under the weight of job cuts, low morale and the demands of federal court oversight.
Criminologists and law enforcement experts said that while they would expect arrests to fall somewhat along with manpower, the trend was troubling. Nationally, arrests were down 11 percent in the same period, the FBI said.
Ah, there's just fewer police officers (and morale is worse) so there's just been fewer arrests. Maybe the Chronicle headline should say “Criminals Getting Away with Crime in Oakland?”
In the Lean Startup approach, Eric Ries and company talk about “vanity metrics” — these are often things that are easy to measure (such as the number of hits to the KaiNexus.com website, recently redesigned) but don't really impact the business and its core (like how many customers and how much revenue does KaiNexus have?). More on vanity metrics here and here.
It seems that “number of arrests” might be a vanity metric for the Oakland police department. How do you measure the safety and quality of life in Oakland?
The talk about vanity metrics reminded me of something I saw in the news recently from Indiana: “Nearly a third of Indiana hospitals had at least one reportable error in 2011.”
Wait, so two thirds of Indiana did NOT have a reportable error? Not reporting the errors doesn't mean the errors didn't occur and we know the underreporting of errors in healthcare is a chronic and serious problem.
The 41 patients with bed sores was the highest number since 2005, when the reports began. But 2011's 17 patients with foreign objects left behind, following surgery, were the fewest ever.
“Reported errors” are down, but does that really mean healthcare in Indiana is safer? 41 patients, statewide, seems like an unbelievably low number. The federal government (the AHRQ) says “pressure ulcers” (bed sores) affect 2.5 million patients per year. Something doesn't add up.
More on the “improvement” (is it?) in Indiana: “Medical errors drop at hospitals.”
When I talked about the high number of hospitals with “zero” errors in Indiana, the Lean Startup folks I was chatting with were appropriately skeptical… can we trust self-reported data? How do we get more transparency about healthcare errors? Or, if Indiana healthcare is REALLY that safe, how do we spread those results around the world?