web analytics
By December 3, 2012 Read More →

Lies, Damn Lies, and Vanity Metrics – in Policing and Healthcare

I arrived in San Francisco yesterday for the Lean Startup conference, where I gave my 5-minute Ignite format talk on the parallels between Lean hospital design and Lean startups – Lean is Lean. There will be video available of the talk online and I’ll share it when I can. Follow the #LeanStartup hashtag or @LeanStartup on twitter for the latest updates from the event.

When I got to my hotel, I saw the front page headline on the San Francisco Chronicle: “Arrest totals down sharply in Oakland.” The article says arrests are down 44%. That sounds great… oh, but wait… is crime actually down 44%??

Nope… from the article:

The arrest figures, obtained under the state’s Public Records Act, raise questions about the effectiveness and assertiveness of the Police Department, which is struggling under the weight of job cuts, low morale and the demands of federal court oversight.

Criminologists and law enforcement experts said that while they would expect arrests to fall somewhat along with manpower, the trend was troubling. Nationally, arrests were down 11 percent in the same period, the FBI said.

Ah, there’s just fewer police officers (and morale is worse) so there’s just been fewer arrests. Maybe the Chronicle headline should say “Criminals Getting Away with Crime in Oakland?”

In the Lean Startup approach, Eric Ries and company talk about “vanity metrics” — these are often things that are easy to measure (such as the number of hits to the KaiNexus.com website, recently redesigned) but don’t really impact the business and its core (like how many customers and how much revenue does KaiNexus have?). More on vanity metrics here and here.

It seems that “number of arrests” might be a vanity metric for the Oakland police department. How do you measure the safety and quality of life in Oakland?

The talk about vanity metrics reminded me of something I saw in the news recently from Indiana: “Nearly a third of Indiana hospitals had at least one reportable error in 2011.”

Wait, so two thirds of Indiana did NOT have a reportable error? Not reporting the errors doesn’t mean the errors didn’t occur and we know the underreporting of errors in healthcare is a chronic and serious problem.

The 41 patients with bed sores was the highest number since 2005, when the reports began. But 2011’s 17 patients with foreign objects left behind, following surgery, were the fewest ever.

“Reported errors” are down, but does that really mean healthcare in Indiana is safer? 41 patients, statewide, seems like an unbelievably low number. The federal government (the AHRQ) says “pressure ulcers” (bed sores) affect 2.5 million patients per year. Something doesn’t add up.

More on the “improvement” (is it?) in Indiana: “Medical errors drop at hospitals.”

When I talked about the high number of hospitals with “zero” errors in Indiana, the Lean Startup folks I was chatting with were appropriately skeptical… can we trust self-reported data? How do we get more transparency about healthcare errors? Or, if Indiana healthcare is REALLY that safe, how do we spread those results around the world?


mark graban lean blog Lies, Damn Lies, and Vanity Metrics   in Policing and Healthcare leanAbout LeanBlog.org: Mark Graban is a consultant, author, and speaker in the “lean healthcare” methodology. Mark is author of the Shingo Award-winning books Lean Hospitals and Healthcare Kaizen, as well as the new Executive Guide to Healthcare Kaizen. Mark is also the VP of Customer Success for the technology company KaiNexus.

book mark graban Lies, Damn Lies, and Vanity Metrics   in Policing and Healthcare lean mark graban consulting Lies, Damn Lies, and Vanity Metrics   in Policing and Healthcare lean

pixel Lies, Damn Lies, and Vanity Metrics   in Policing and Healthcare lean
pinit fg en rect gray 28 Lies, Damn Lies, and Vanity Metrics   in Policing and Healthcare lean
Please consider leaving a comment or sharing this post via social media.

5 Comments on "Lies, Damn Lies, and Vanity Metrics – in Policing and Healthcare"

Trackback | Comments RSS Feed

  1. Mark Graban
    Twitter:
    says:

    Here are my live notes from the conference via Google Docs:

    LINK

  2. Mark Graban
    Twitter:
    says:

    Eric Ries kicked off the event saying “no success theatre and no vanity metrics” from the presenters. I love it.

  3. Hi Mark. “Vanity metrics” evolve because of the wrong incentive systems, in my opinion. (Maybe root cause analysis needs to happen.) Thanks for drawing attention to this practice and giving it a name. How do we stop its use?

    • Mark Graban
      Twitter:
      says:

      Giving credit where it’s due (other than me), the term “vanity metrics” isn’t something I coined. It’s in wide use in the Lean Startup community. I think Eric Ries coined the term.

      How do we stop its use? I dunno… in an organization, leaders (and boards) need to make sure people aren’t just reporting the things that make them look good. We need critical thinking to drill into the metrics. If you’re a hospital CEO and people are saying “we didn’t make any medical errors last month” then GO AND SEE. Is reality the same as the data?
      Mark Graban recently posted..Lies, Damn Lies, and Vanity Metrics – in Policing and HealthcareMy Profile