Sponsored by the book "Lean Hospitals" | Free Download of First Chapter


Wednesday, July 02, 2008

Gaming the System: E.R. Targets

The BBC morning news show had a story about what they call "Accident & Emergency" here in the U.K. There's a lot of discussion, generally, in the news about improving true service quality instead of relying on targets.

One example given in the BBC story... there is a 4-hour target, where patients must be seen within four hours of arriving at A&E... and there are cases where, after 3 hours and 58 minutes, the patient is admitted to a ward. Problem solved, right? Well, except for the extra cost and risk involved in being admitted.

That's an example of gaming the system -- a fake improvement that suboptimizes one metric over the entire system.

This older BBC story, from 2006, talks about similar dynamics:
But they also said they had led to gaming, citing the four-hour A&E target which had meant extra staff were brought into casualty when performance was being measured, meaning operations elsewhere in the hospital had to be cancelled.

Another gaming method involved patients having to wait in an ambulance outside A&E until staff were confident they could meet the target.
I'm not bashing the NHS employees... their behavior is perfectly rational behavior. Given targets, people are really creative about hitting them. That's why we need real systemic process improvement, not more targets.

While I'm here, I'm hoping to meet John Seddon, the guru of moving away from targets and command-and-control management.



Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board

Labels: , , ,

Sunday, May 04, 2008

Obvious Incentives for Coverups

The Associated Press: FAA tries again to fix cover-up of air safety errors:

It's always disturbing to hear about cover ups and systemic problems related to aviation safety, whether it's airplane maintenance or air traffic control. In this case, it's the controllers:
"... the Transportation Department's inspector general found FAA managers in Dallas-Fort Worth routinely and intentionally misclassified instances where airplanes were allowed to fly closer together than they were supposed to, the FAA said. Instead of calling them operational errors or deviations from safety rules by FAA controllers, the managers labeled them pilot errors or nonevents."
Yikes, there had better be some consequences for that. Something sure is rotten within the FAA.
I can't find the original local article I saw in the paper on Saturday, an article which posed a question of basically "well why would people do that?" The incentive seems pretty obvious... gaming the numbers to look good and to avoid punishment for controller errors.

This article sums it up:
By masking the mistakes as pilot errors, he said, the controllers and their managers were able "to escape accountability."
It sure would be nice to see an FAA culture develop where we're not relying on whistle blowers coming forward to report systemic rot like this.

A special counsel investigator found:
"I continue to be concerned about a national trend," Bloch said in a statement referring to the Dallas-Forth Worth cover-up and the recent disclosure of lax FAA supervision of safety compliance by Southwest Airlines and American Airlines. "These problems exist because of a culture of complacency and cover-up in the FAA. This culture did not develop on its own. I believe it happened with the complicity of higher management and could not have been possible without the support of leadership in Washington."
So I guess we can't blame individual controllers for mischaracterizing the errors? Is part of the systemic problem a shortage of controllers? That's been a complaint at O'Hare (where there are many near-miss incidents each year) and it's a complaint at DFW:
The air traffic controllers' union, deep into a two-year-old fight with the FAA over manpower and safety, pounced on the agency's announcement to again criticize what it considers a shortage of workers. The Dallas-Fort Worth facility has 57 fully certified controllers, down from 99 in January 2006, said Darrell Meachum, vice president of the National Air Traffic Controllers Association's southwest region.
Are they understaffed now or were they overstaffed before? We can't tell from the outside. But being chronically understaffed could certainly create conditions where errors are more likely to occur... and a culture of cover ups... sheesh, that doesn't help, does it?

Why are so many organizations prone to this cycle of cover ups and blame? Why can't leaders "embrace their problems," being open about waste and working together to solve problems and prevent process defects? Expecting too much out of people, huh?

Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board


Labels: ,

Tuesday, March 04, 2008

Deming on Education

If there was any value to the Industry Week commentary that tried to link Deming and Obama, it was that it prompted me to re-read some of Deming's work, namely the "management of people" chapter from The New Economics.

There's so much good stuff in there, including the 14 rules / guidelines for a manager of people to follow. You can read that online at google books (for free). There's so much that's re-stated, from Deming, in many of the more recent books about Toyota. You definitely see the Deming influence coming through. So it's sometimes interesting to go back and re-read earlier books.

In the chapter, Deming rails against business schools, pointing out what they SHOULD teach, which is, of course, the opposite of what's taught. Deming says business schools should teach students about the "evils" of short-term thinking and the "evils" of the merit system and ranking people. There's also a somewhat bleak chart on page 122 that makes the case that schools and management systems do nothing but demoralize people throughout their lives until they die.

Deming then, on page 145, rails against grading students and grading teachers or schools. Deming's argument is that grades (especially forced ranking and grading curves) rob students of their intrinsic motivation to learn (and probably robs teachers of their joy in teaching).

Deming recommends:
  • Abolish grades (A, B, C, D) in school...
  • Abolish merit ratings for teachers
  • Abolish comparison of schools on the basis of scores
  • Abolish gold stars for athletics or for best costume
He writes, "Our schools must preserve and nurture the yearning for learning that everyone is born with."

In recent years, the trend has been toward "merit pay" for teachers and schools. Hogwash. Deming, the hypothetical presidential candidate, would undoubtedly be against the "No Child Left Behind Act" (but maybe for different reasons than Democrats).

From the wikipedia page:
NCLB is the latest federal legislation (another was Goals 2000) which enacts the theories of standards-based education reform, formerly known as outcome-based education, which is based on the belief that high expectations and setting of goals will result in success for all students
High expectations and goals without a method? That's a recipe for failure and I assume Dr. Deming would have hated that. The focus is on measurement... but at the expense of learning? Given goals, people in any setting are clever about "gaming the numbers" (as the Wikipedia article points out) and educators are no different.

I'm not a NCLB expert... reading more, I'm guessing Dr. Deming wouldn't disagree with the whole act. Making sure that teachers are well qualified is a good thing. All things considered, I guess that Dr. Deming would suggest "leadership" as a replacement for NCLB. What do you think?

Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board


Labels: , ,

Monday, January 14, 2008

Gaming the System at Starbucks

Starbucks Gossip: Howard Schultz apparently hasn't put the brakes on warm sandwiches

Following up on last week's post on Starbucks operations, I found an example of "gaming the numbers" in a different post.

"Gaming the numbers" is a common phenomenon when people are pressured to hit certain numbers (a dynamic that Deming preached against, this idea of "management by objectives" being dysfunctional). I'm often reminded of the three things that can happen when people are pressured to hit numbers (credit to Brian Joiner):

  1. They can distort the numbers,
  2. They can distort the system, or
  3. They can improve the system.
#1 and #2 are usually much easier to do. Click on the "Gaming the Numbers" link at the bottom of the post to see more examples.

Apparently, Starbucks has a staffing/labor formula that's based on workload and volumes. Some stores have new warming equipment (for the breakfast sandwiches some stores are selling) that can also be used for warming muffins or other pastries. At least one employee found a way to distort the system:

It also helps, though it's cheating a little, to press the warming button for EVERY pastry, even when you don't warm them. We ear 1 hour of labor for every ten times we push it per hour during peak. And believe me, that adds up.
Ah ha, they've found a way to get more labor allocated or budgeted.

Another employee posted a warning:
Adrienne, just to warn you...The P&AP dept will be looking for that trend in stores and it will be treated as time theft. Please learn a different way to increase labour without "cheating" you could get into trouble!
The original poster wrote back and said it was her District Manager who had shared the idea!!

Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board


Labels: ,

Thursday, December 20, 2007

Gaming the Numbers in the UK, Again

Blundering hospitals 'kill 40,000 a year' - Britain - Times Online:

Ah, gaming the numbers... everyone's favorite pastime. It certainly is here in the State, even though I have my second story of the year related to gaming the numbers in the UK health care system. The first was a story about ambulances fudging and cheating their way to making the numbers for on-time response.

In this story, hospitals are gaming the system by underreporting patient safety incidents.
"...the number of mistakes to which NHS hospitals openly admit is a small fraction of the total accepted by the Government’s patient safety watchdog.

It found that only 276,514 errors were recorded each year by English hospitals, even though the National Patient Safety Agency (NSPA) puts the true figure at closer to 900,000.

... almost 10 per cent of the trusts surveyed claimed an unlikely error rate of zero."

That would be quite a headline if they really had gotten infection rates to zero. Some hospitals are making great progress on that front (with methods including Lean), but zero is darn hard to get.

People underreport for many reasons - pride, fear of being sued, not wanting to admit problems. Compare that to what I heard about during my tour of Virginia Mason Medical Center. At VMMC, one of the leaders in the use of Lean, they were happy that internally reported patient safety incidents had gone up. Had the hospital suddenly gotten much less safe? Of course not. It meant that people were finally reporting errors, which was the first step in finding root cause fixes and prevention.

It's something I discuss in my upcoming book -- the need for more openness and sharing of information, both inside hospitals and across hospital boundaries. Back to work on that thing now... I'm learning you're never really "done" with a book... you just stop working on it at some point!!!

Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board


Labels: ,

Tuesday, November 20, 2007

NUMMI Employee Says Plant Hides Defects

Whistleblower says defects hidden at Toyota-GM Calif. plant

Ironically enough, while on the topic of not counting defects (the TSA), this disturbing story was in the Detroit News today. A 23-year employee, a quality inspector, has filed a lawsuit claiming that the company mistreated her when she reported defects.

"According to legal documents obtained Tuesday by The Associated Press, defects that were intentionally passed over included broken seat belts, faulty headlights, inadequate braking, mirrors falling off, engine oil leaks and steering wheel alignment problems -- all in an effort to decrease the number of defects. It is not clear whether any defects resulted in accidents.

When Cameron, a trained expert at spotting defects, complained, her bosses struck back, demoting her twice, accusing her of being crazy and violent, forcing her to submit to mental fitness tests, according to the documents. "

I certainly hope none of that is true. As the article points out, that isn't supposed to be Toyota behavior. It doesn't illustrate the ideals of quality and teamwork, nor does it seem to represent empowered employees who are able to stop the line.

I don't know what is going on inside NUMMI, by any stretch. My earlier posts about a plant tour I was on two years ago sometimes bring random and angry comments from people claiming to be NUMMI employees. The quality reputation of NUMMI is good, as judged by outsiders, not just from internal quality reports. Is the story overblown here or is NUMMI "the best of a bad bunch," where things like this are just commonplace for the industry? I certainly have my own first hand experience at an auto parts plant where top managers bent the rules on quality to keep the line running, although I never heard of falsification of quality records.

Is Toyota perfect? No, they are a company full of people, and we are fallible, we make mistakes. That's not an excuse for any of the alleged behavior, if it is true. If stories like this are true, that's bad, for the people involved, and for anyone trying to use Toyota as an inspiration for trying to do things differently, in terms of quality and employee relations. Are people at Toyota capable of those things? Probably. Is it widespread throughout the company? I hope not. Does stuff like that happen at "mass production" companies? I'm certain of it. But we hold Toyota to a higher standard -- they have asked to be, right?

While NUMMI is "jointly managed" by GM and Toyota, I consider it a Toyota plant under the Toyota Production System. It's jointly owned, but the burden needs to be on Toyota, I'd say.

It will be interesting to see how this case plays out. Here are more stories via Google News. Here is a post at TheTruthAboutCars.com that includes an alleged detail:

Cameron began retaining her original pencil written reports (to document the changes) and turning-in photocopies.

The blog has reader comments, including this story (from another auto factory) about fudging the quality numbers.

The SF Gate has a story on this, as well.

Subscribe via RSS | Lean Blog Main Page | Podcast | Message Board

Labels: , ,

Thursday, June 07, 2007

Making the Numbers

Freedom from Command & Control: Rethinking Management for Lean Service

Here is a fascinating post from John Seddon at his website www.lean-service.com about ambulances "making their numbers":
The stupidity of measuring the eight-minute response time has been all over the press recently (you get a tick in the box for getting there in eight minutes even if the patient dies). People working in the ambulance service tell me this is causing some managers to send ‘technicians’ who get there on time but can’t help the patients (some die) and other managers work on what calls to ‘exclude’ from the eight-minute measurement. The latter, of course, leaves them open to making the mistake of treating some events as not time-critical when in fact they are.

Sigh, that's familiar from the manufacturing world and the business world. Donald Wheeler talked about (quoting Brian Joiner, I believe) the three things people can do when pressured to improve the numbers:
  1. Distort the system
  2. Distort the numbers
  3. Improve the system
In the case of the UK ambulances, we have the first two happening. They distort the system by sending SOMEBODY in the 8-minute window. They distort the numbers by looking for cases to exclude for one reason or the other. Neither is a true system improvement, unfortunately. That's what we need more of in this world, actual system improvements, not distortions.

Here are some BBC stories on the 8-minute goal:

BBC News Story

Article 2

Labels: , ,

Sunday, May 27, 2007

Freakonomics and Lean

Freakonomics: Official Site

I know this wouldn't count as "leisure reading" for many folks, but I'm reading (and very much enjoying) the book Freakonomics: A Rogue Economist Explores the Hidden Side of Everything. I'm a nerd, I read books like this for fun, I'm fine with that. Stick with me, since there are some applicable lessons for the Lean world, if you'll accept my summary and shared examples.

The link above is to the authors' official web page, where they offer to sign your book, in a pretty Lean manner (this would be a nice challenge for other authors to follow). Instead of the waste involved with shipping the book to the authors and the authors sending it back (with autograph), if you request it on the web page, they'll both sign a sticker, a bookplate, and they'll send it to you at their expense so you can put the sticker in the cover of the book or "on your forehead" if you want. Very nice gesture on their part, and waste reducing (not that FedEx, UPS, and the USPS would appreciate that revenue-stealing plan).

Back to the main content of the book. In the first chapter, Levitt and Dubner, give many examples of using incentives (or attempts to use incentives) to drive positive behavior or avoid negative behavior. That's a topic that should be of interest to Lean change agents and leaders -- we're always trying to drive the right behaviors, such as "how do we get more employee suggestions?" or "how do we get people to follow the standard work?"

The authors layout three different types of incentives: economic, social, moral. In business, we tend to focus on economic incentives. In fact, saying "economic incentives" may well be redundant in most companies. We pay bonuses for hitting sales quota numbers, that's an economic incentive. We threaten to fire someone for not following safety procedures, the potential loss of one's paycheck is an economic incentive as well.

Let's look at something like wearing safety glasses (or, in a hospital, clinicians washing their hands). These are practices that should be followed 100% of the time, but aren't. An economic incentive might pay some sort of bonus for compliance or a penalty for non-compliance). A social incentive might involve some sort of social shame for non-compliance -- is it somehow embarrassing to be caught doing the wrong thing? A moral incentive plays on the "it's just wrong" aspect of human nature.

Do incentives work? Sure, in theory, but the theory is also complicated (and the reality of incentives is complicated) because of human psychology. That's why, I believe, Dr. Deming taught us that understanding employee psychology was a critical aspect of management and leadership. The world isn't so simple so that we can rely only on smartly-designed economic incentives.

Back to Freakonomics:
"For every clever person who goes to the trouble of creating an incentive scheme, there is an army of people, clever and otherwise, who will inevitably spend even more trying to beat it."
Very true, don't you think?

The book gives an example of a study conducted on day care centers. One of the parent behaviors that the day care centers want to discourage is late pickups, since this prevents the center from closing and increases their costs. In the study, the percentage of late pickups was measured for a few weeks. Then, some centers instituted a new policy of a $3 fine to any parent who was over 10 minutes late.

What happened with the penalty, the incentive for coming on time? Lateness went up!!!

Why? The Freakonomics authors say that the old moral or social incentives (embarrassment or guilt for being late) was now replaced by an economic incentive. $3 was hardly a huge penalty compared to the $360 monthly tab. Parents no longer felt guilty (the moral incentive) because they were now paying a trivial financial penalty.

That's quite an eye opening lesson. I don't know exactly what the answer is for challenges like safety glasses and hand washing. As we've discussed before, I would tend to rely more on direct observation and leadership more so than financial incentives that are easily ignored or easily gamed.

Labels: , ,

Wednesday, February 21, 2007

Another Sad Example of Gaming the System

Back to my original post about gaming the system, there are three ways people can respond to pressure to improve:

  1. They can work to improve the system
  2. They can distort the system
  3. They can distort the data
I got an email from an old friend -- I'll keep his name, company, and industry confidential, but he said I could share this story. It's a major company with a long history, a company that should really know better, but they've lost their way, apparently.

"We have a position here called validation engineer. They are responsible for getting all the components of a program to 'pass' all of the required functional tests by a certain date set by program managers. They have no design input or ability to change it, a limited budget, and low part familiarity."

So the first thing that jumps out is the "job description" or purpose for that role. I'm sure it's not the official description... but you'd think someone responsible for testing (or inspecting quality) has the right and authority to either "pass" or "fail" an item. Their job shouldn't be to pass items -- or why have the validation job at all? Is it the role of a production inspector to pass everything? Of course not.

The fact that a program manager "has" to hit the date requires that they get designs that can pass the validation, not to pressure the validation process to be skewed or rendered worthless.

"Literally when we find a failure their response has been 'we can't report that, I need the part to pass or I'll be fired'. Needless to say there is high turnover rate and they are all hunting for jobs right now but the impact on every part of product development is affected."

There's the example of gaming the system... are people being pressured to fudge results or to cheat? Of course they are. Since employees aren't allowed to work with integrity, no wonder they get frustrated and want to go to a job where they can work with integrity. It doesn't rank high on the "respect for people" scale when you force them or pressure them to give up their integrity in exchange for a paycheck. If people find other job options, they'll take them. I really feel bad for the employees who are placed in this position.

My friend later admitted:
"They addendum is that I am frequently asked to lie. Often I have to make a judgment call. My integrity is the price that I pay, but I always try to remember that deception is only a short term gain. Someone may lose their job immediately but we'll may all be out of business if recalls are too high and we lose business. Or the other case, they go to another outside test lab who will give them a pass."
No wonder my buddy has been figuring out how to find a new job or a new industry, and he's been learning a lot about Lean (which only leads to more frustration with his non-Lean environment, although he's taking steps to make improvements).

Sad situation. Management is actively encouraging the distortion of the system and the distortion of the test results rather than working to improve the quality of the product designs. No wonder this company is struggling. They have to do better. When my friend gets a new job, maybe I can blast the company publicly, by name.

Labels: ,

GM Got Gamed

Following up on my post about my recent experience metrics and processes being distorted (and my less-than-perfect lean coaching efforts), I was thinking about to some first-hand experience I had at a GM engine plant, circa 1995. It's the most blatant example of someone intentionally distorting data that I've ever seen.

Our engine block line was designed at a throughput goal of 92 blocks per hour, if everything ran perfectly (but it hardly ever did, at least not for an entire 60 minute stretch). Our plant superintendent, Bob, (the #2 guy in the plant) decided that 60 pieces per hour was an acceptable number (partly based on productivity numbers that were attributed to Toyota). Anything below 60 and you’d have to explain why. Now, he wasn’t really the listening, problem solving type. He managed by fear, yelling, and intimidation. There was more yelling involved than listening or problem solving.

Anyway, at the end of the engine block line was a mechanical counter that recorded the hourly production counts. The UAW workers who unloaded blocks dutifully recorded the number every hour.

The numbers might typically look like (for you RSS readers, you might not see the image):


That's an average of 56.25 pieces per hour. Not quite up to Bob standards, although we exceeded the goal in 4 hours and came close to 92 in one hour.

At the end of the day, before our “4 o’clock meeting” where the plant salaried staff took its daily verbal beating from Bob, Scott, the production supervisor (technical title of “Team Coordinator” didn’t quite fit) would pick up the counts and do a little daily editing.

He’d take the numbers and turn them into, I kid thee not:


That's still an average of 56.25 per hour. But it's much more consistent, isn't it?

Bad ole’ Bob never questioned these numbers. I know it's hard to believe that he would believe those numbers, but when reviewing multiple departments, Scott's fudgery helped avoid too much attention that a really bad hour would bring upon him. Rather than asking "why don't we have more hours of 86 blocks?" the upper limit of expectations was set too low, at 60. I asked Scott once why he fudged the numbers each day, and his answer was:
"Bob wants 60 an hour, he gets 60 an hour."

Other departments got more than their share of the daily beatings (I had a bet with a co-worker if Bob would say “pathetic” or “miserable” first). Bob always had the same pronouncement for our problems: we weren’t trying hard enough. And apparently, more yelling from Bob was what we needed to motivate us. But that never worked.

“Not trying hard enough” fell into two categories: 1) urgency and 2) intensity. We didn’t have a sense of urgency. We didn’t have the proper intensity. Like a shorter imitation of Mike Ditka (with a signature bad toupee rather than a signature mustache), Bob would yell and scream and spit. Sometimes we got “we need urgent intensity” or “we need intense urgency” if things were really bad. All of the yelling and screaming, all of the fear, all of the fudging of the numbers got in the way of true process improvement and true problem solving.

Obviously, situations like this are part of the reason our plant manager eventually got moved out of the way for a new, NUMMI-trained plant manager. That started our road to recovery, as a plant. It was never a worker problem, it was a management problem. That's an important lesson of lean -- what's required is a change in management practices and management philosophy.

I'll leave it for another post to talk about that "4 o'clock meeting" and what its goals were supposed to be. The meeting was designed by some internal lean consultants we had, but was co-opted for non-lean management methods. Why weren't the lean consultants being listened to? Again, I'll save that for another post.

Illustration from igotzillustration.com

Labels: ,

Tuesday, February 20, 2007

We Got Gamed

A somewhat recent experience prompted me to re-read portions of Donald Wheeler's outstanding book, Understanding Variation: The Key to Managing Chaos, with its section on the three ways people can improve a metric or a target (he gives credit to Brian Joiner):

  1. They can work to improve the system
  2. They can distort the system
  3. They can distort the data

I’ve always observed that people are incredibly clever in figuring out how to game a system to their advantage. I’ve seen it (in a very positive and worthwhile way) while observing an outpatient cancer treatment center, where repeat patients learned which appointment slots tend to get delayed the least (patients learning how to NOT waste time waiting for cancer treatment, you can’t argue with that). The patients’ actions aren’t purposely delaying other patients, so nobody is being harmed.

In most cases, though, somebody or some organization is being harmed by more selfish gaming of the system. It’s sometimes easier for people to distort the system or to distort the data in order to get rewards, be it a pat on the back or an annual bonus check.

One time, with a client, we were measuring the batch sizes of specimens being delivered to the laboratory. Batching of specimens (drawing blood from 10-15 patients before sending it to the lab) was a big contributor to “turnaround time” (cycle time) for the lab. We thought we had explained how we wanted specimens delivered in smaller batches, even if that meant it took more time to do the blood draws and specimen collections. We were measuring the size of the batches for a while until somebody tipped us off to the distortion.

Some of the phlebotomists (employees who draw blood) were taking a batch of 12 patients' specimens and shipping it down to the lab in four sequential batches of 3 patients each. This made the metric look good, but the lab was still essentially getting a batch of 12. How embarrassing, I thought. I was embarrassed that we hadn’t done a better job of explaining “WHY” we needed smaller batches. Since employees were gaming the system, I felt like there was a failure on my part.

Before jumping to the conclusion that people are being malicious or don’t care, I try to stop and think how their actions might be reasonable. Someone on our team said, “maybe they didn’t understand that it was a turnaround time issue, maybe they thought the tube system physically couldn’t handle large batches?”

Rather than beating up on the phlebotomists, here are our next steps:

  • Re-iterate to them WHY we need smaller batches,
  • Explain what “true” small batches really are and how that benefits the patients
  • Change our measurement system so we track not only the sizes of batches that arrive, but who they came from and at what time they arrived (so we can detect “fake small batches”)
  • Emphasize to the phlebotomists that they need to make the extra trips to the tube station to send the smaller batches. We need to make sure that the lab will pay for enough resources to get the job done with the desired cycle time.
  • Do more direct process observations and Standard Work audits to directly see if phlebotomists are following the correct process or not.
One of my own "kaizen" steps for the future will be to "FMEA" a new metric, to think through how people might game or distort the metric so we can avoid that kind of behavior.

Labels: , ,

For more posts, click here for the LeanBlog Archive

Search the LeanBlog and the rest of the Lean "Blogosphere"