Is the Drop in MLB Attendance a Real Signal or Just Noise?

TL;DR: Headlines about declining MLB attendance focus on short-term percentage drops, but those comparisons lack context. When attendance data is viewed using Process Behavior Charts, most year-to-year changes appear to be normal variation. Only when data falls outside expected limits does it make sense to ask what truly changed in the system.

Headlines often claim that MLB attendance is falling sharply or hitting multi-year lows. But without historical context, those claims can be misleading. Using Process Behavior Charts, this post examines long-term MLB attendance data to determine whether recent declines represent a meaningful signal or just normal year-to-year variation.

My book Measures of Success has some methods and mindsets that might be useful to you.

Why Attendance Headlines Can Be Misleading

This past week, there were headlines about Major League Baseball attendance, including:

MLB Attendance Drops to Lowest Average in 15 Years (Fortune)

The article says:

MLB attendance has dropped to its lowest average in 15 years, down 6.6% from this time last year and 8.6% overall, according to Stats LLC. The league could see its first season since 2003 with average attendance  below 30,000.

Why Two-Point Comparisons Aren't Enough

There's a risk that we can be misled by text-based comparisons of two data points. The Fortune article tries to set additional context by saying this could be the first season below 30,000, but, again, is that number statistically meaningful?

Should MLB be worried? Is it worth the effort to start brainstorming the causes of the apparent decline? Should people be talking about countermeasures or how to improve the game? Or, do we run the risk of overreacting?

The best and most practical method I've used is called a “Process Behavior Chart” (a term coined by Donald J. Wheeler, who wrote “the book that changed me,” Understanding Variation and also contributed a foreword to my book).

How does this method work? How do we analyze similar data in the workplace?

Why Tables and Simple Charts Hide Important Context

First, we have to get the MLB attendance data. Baseball-reference.com, as always, is a great source. However, tables of numbers are not the best way to look for trends, as illustrated by the MLB data, as shown below, at left. Far too often, so-called “dashboards” at work display endless and unhelpful tables of numbers, as also shown below, at right.

Another thing we shouldn't do is to visualize just two data points using a Column Chart with a misleading Y-axis and a breathless comparison:

Again, we don't know if a 7.25% change is a “signal” of a significant change in the data. It could just be “noise” in the data that's not worth overreacting to.

What a Run Chart Shows Over Time

A Run Chart is more helpful for providing context. This chart includes data going back to 1980:

Over time, attendance has increased. This year's attendance is 36% higher than 1980, if we want to play the two-data-point-comparison game. Attendance is down 14% from a decade ago. We can also see what appears to be a big drop in attendance in 1995 (after the World Series was canceled in 1994).

The Run Chart and almost 30 years of data provide additional context that newspaper headlines and simple comparisons cannot.

Turning the Run Chart Into a Process Behavior Chart

The surest way, however, to determine if MLB attendance is a “predictable system” is to turn the Run Chart into a Process Behavior Chart. To do so requires a few simple calculations (as you can see in this spreadsheet and this PDF handout about the methodology).

Using the first 25 years of data from 1980 to 2014 as the baseline, we calculate the average (the green line) along with Lower and Upper “Natural Process Limits” (the red lines). This forms what we call the “X Chart” in the Process Behavior Chart methodology (there's also a companion chart call the “MR Chart,” but we'll leave the detail out for now).

In a predictable system, a measure would tend to fluctuate around the average, within the calculated Natural Process Behavior Limits. Since MLB attendance has been increasing over 30 years, this chart shows we don't have a predictable system or measure over that timeframe. Attendance looks pretty stable, though, over a more recent period.

Here is an example of an X Chart for a different metric that shows a predictable system, with results that fluctuate between the calculated Natural Process Limits:

When we have a predictable system, we can, well, predict that future performance will be between those limits — unless something changes in our system. Each system or metric we track has its own inherent routine variation. A 7.25% change in one system might be noise (if the typical year-to-year variation is 10%), while it might be a signal in another. With my blog chart, above, it would have been a waste of time to ask why the number of page views was higher or lower in any given month. Every data point in that chart is just noise.

When a Drop Becomes a Statistical Signal

Back to the MLB data, if we draw an X Chart and calculate the average and limits using 2009 to 2017 as a baseline, we do see a predictable system over that timeframe:

The Process Behavior Chart would have predicted that 2018 attendance would fall between 29,680 and 30,278 unless something changed.

2018's number, so far, falls below that Lower Natural Process Limit, so it is indeed a “signal” that tells us something has changed:

What a Signal Does–and Does Not–Tell Us

The chart tells us something changed in 2018, but it doesn't tell us what changed. That signal, a point below the Lower Limit, does mean it's worth trying to understand what happened. Edit: Chad Walters raised a great point in a comment that we're not comparing apples to apples. The other years are full-season numbers. The 2018 data is a partial season. Does attendance usually go up in the summer months? I'd guess so.

This article said:

Heading into the weekend, the average attendance of 27,207 was down 6 percent from 28,931 on the Friday before Memorial Day last season. Last year's final average of 30,042 was the second straight drop,

That's more of an apples-to-apples comparison but it's still just two years.

Since 2016 and 2017, attendance numbers were “noise” or part of routine variation in the metric; asking “what happened?” in those years would have been a waste of time. a “second straight drop” doesn't mean that's a statistically meaningful trend. A predictable system and metric will sometimes do that as part of routine fluctuations.

The Management Lesson: Reacting to Metrics the Right Way

As Wheeler says, one error managers can make is to overreact to every up-and-down in a metric. Asking people to explain every decline or every below-average data point isn't a good use of our limited time. There is no “root cause” for noise in a metric.

We do need to make sure we react to signals, as we see in the 2018 data. Now it's worth speculating about weather, competitive imbalance, too many strikeouts, or other alleged causes of the drop in attendance. What changed here in 2018?

When looking at a workplace metric, we need to learn how to separate signal from noise. The Process Behavior Chart allows us to do that. When we see a signal, it's appropriate to ask “what happened?” in a reactive way. Even if a metric just shows noise, we can still improve. We just need to go about that in a more systematic way.

Instead of asking “what happened?” we can step back and ask “how can we improve the system?” This is where Lean methods like “A3 problem solving” can help.

So, wasn't this just a really wonky, roundabout way to prove that MLB attendance had dropped? I don't think the exercise was a waste of time, since we didn't know that 7.25% was an extraordinary variation until we looked at more data. Without that context, a 7.25% drop might have been consistent with past year-to-year variation. We don't know for sure until we chart the data and calculate those Natural Process Behavior Limits.

What do you see in your workplace? Do you rely too much on simple comparisons of two data points? Are you expected to overreact to the data? Does doing so consume valuable time that could be better applied to improving the system instead of trying to explain it?

Instead of reacting to every up or down, leaders need methods that tell them when a question is worth asking. Process Behavior Charts provide that discipline–whether the data comes from baseball or the workplace.

---

Please scroll down (or click) to post a comment. Connect with me on LinkedIn.

---

If you’re working to build a culture where people feel safe to speak up, solve problems, and improve every day, I’d be glad to help. Let’s talk about how to strengthen Psychological Safety and Continuous Improvement in your organization.