Debunking a Misleading Minimum Wage Study

Dave Thompson

The National Employment Law Project (NELP) released a study claiming to have directly disproved any link between minimum wage and job loss. Unfortunately, it has received media attention, including a piece or two by Nick Hanauer, “America’s worst minimum wage pundit.” Fortunately, at least some of the coverage is negative – as it should be. The NELP study is not serious, and it’s also flat wrong.
Numerous people have pointed out the primary issue: NELP asked the wrong question. Asking the wrong question is a common but serious error, and can invalidate your result. Here’s a revealing sentence from the report:

“To the contrary, in the substantial majority of instances (68 percent) overall employment increased after a federal minimum-wage increase.”

While they claimed to look for correlation, they didn’t. For that sentence to make sense, their question must have been: “Is a raise in minimum wage more likely than not to cause a reduction in total jobs?” Their answer is no. But is that the usual meaning of “kill jobs”?
The Wrong Question

I grew up on a farm, and we had barn cats. These cats were mostly strays, mostly wild, and all mortal. If we had ten cats in 1997 and eleven in 1998, can we say no cats died in 1997? Of course not. All we can say is that more arrived or were born that year than died, full stop. Something could have still killed one, two, or ten that year.

This is a lot like jobs. The U.S. population hasn’t gone down since 1918. So it’s rather neat that overall employment hasn’t gone down often. Instead, over the 77 years in question, the workforce has gone from 26 million to 120 million jobs. If average growth was 0%, we’d be at >80% unemployment instead of ~5%. If we want to see if something is killing jobs, we need to ask a better question.
The best question would be “Are growth rates different after a minimum wage increase compared to other years?” But to compare to the NELP study, allow me to ask a more restrictive question: “Does negative growth (job loss) occur more often after a minimum wage increase than in other years?”
That answer, based on NELP’s yearly data, is unequivocally “yes.”
What the Data Actually Show
We've got data for 36 years, although ‘97 and ‘98 aren't usable. Of the 34 usable years, only 9 are after minimum wage changes. Of these, 3 show job loss in the restaurant industry (1991, 2008, and 2009) - that's 1 in 3. Of the 25 years with no minimum wage change, none show job loss in that industry. None.
If your friend wins 25 coin tosses in a row and then you lose 3 of 9, you can safely bet that he's using a different coin. One statistical test for this is Fischer's Exact Test, and here it shows that the difference between the two cases (3 of 9 vs 0 of 25) is statistically significant. The same can be shown for retail: job losses are seen in 6 out of the 9 years after minimum wage changes. Job losses are seen in only 2 of the remaining 25 years. Total employment decreases in 4 of the 9 years with a minimum wage increase, whereas such decreases are seen in only 2 of the other 25 years.
However, most of NELP’s reported findings are based on the monthly data, which are more complicated. I can’t use Fischer’s test here because the monthly data are not independent – they overlap. The years weren’t truly independent either, but at least they didn’t overlap. So instead, I’ll try to address the first question: are the growth rates different?

Highlighted areas indicate key differences: there are proportionally more bad (2% or more job loss) periods when the minimum wage is raised and fewer strong growth (2.5-5%) periods. The only data contradicting the trend are those reflecting greater than 8% growth. Every month in that range comes from the minimum wage raise that occurred in October 1945. Astute readers may realize that the return of millions of GI’s and the end of war rationing may have had some important effects. Excluding that year, the average growth is 1.75% in years without a change in the minimum wage and only 1% with. That may not sound like much, but if the economy grew at those two rates for 77 years, that would be a difference of 44 million jobs. More than one in three of our 2015 total.
Correlation and Causation
A note of caution: I’ve shown that minimum wage raises are associated with job loss. I’ve shown that retail job growth is slower after minimum wage raises (overall job growth is as well). What I haven’t shown is causation. Studies looking at causation have shown mixed results – some show significant differences, some don’t. That's not terribly surprising given the vast number of confounding factors involved. If you read these studies, keep in mind that “finding no statistically significant difference” is not the same as “proving no difference exists” – small datasets often cause the former and we don’t even have tests for the latter.
Those provisos aside, I hope I’ve shown that the simplest meaningful analyses indicate minimum wage may very well be killing jobs, contrary to what NELP found after asking the wrong question. That minimum wage has an effect on employment also aligns with a mainstream understanding of elastic supply and demand, as well as thought experiments of $500/hour minimum wages. NELP claims opponents of minimum wage policies are driven by ideology rather than evidence. I hope I've provided some good reasons to doubt that claim.