Fun with Numbers or “Lies, Damned Lies, and Statistics”

In my work we use statistics to support or disprove a point. We mine data to look for patterns, answer questions, and search for evidence of impacts. We try very hard to be objective and accurate in our use of statistics. I spend a lot of time reviewing work of others and must be on guard against biases and mistakes in using statistics. Time after time I have asked authors and analysts to go back to the data to take another look – to look at the data from a different perspective, to slice and dice it in different ways to see if a different story emerges. In the process I have grown to appreciate the beauty and danger in data – they can be used and misused in a variety of ways. Sometimes they are misused on purpose, sometimes by accident, sometimes because our biases lead us to look for patterns we believe must be there.

Two articles I read recently led me to start this post as a way of providing examples of the misuse of statistics that are occurring in current political dialog. The first I described in my previous post on measuring educational achievement. Blogger IowaHawk examines ACT/SAT scores and demonstrates that if you do not think clearly about the numbers they can lead to faulty conclusions. He compares scores between Wisconsin and Texas. Wisconsin, with high union involvement in education, has an average SAT score significantly higher than non-union Texas. However, IowaHawk shows that comparing SAT scores for people of the same ethnicity produces very different results. Texans of the same ethnicity tend to score higher on SAT scores than those in Wisconsin. (Which pains me to write, given I live in Wisconsin.) Texas is home to significantly more minorities than Wisconsin – and that is the driver of the average SAT score difference. IowaHawk concludes “In other words, students are better off in Texas schools than in Wisconsin schools – especially minority students.”

The second article was a blog post by Russ Roberts on CafeHyek. Much fussing has been done lately about the plight of the middle class. One bit of that fuss has been about the apparently meager progress made in middle class incomes over the past few decades. Median household income in North America did not change much between 1980 and 2005. On the face of it, that sounds like a bad story – particularly when you place that against the high profile high earners that we read about in the paper including from bankers, Wall Street types and professional athletes. However, Roberts demonstrates that we ought to dig deeper into the story before we draw the conclusion that we have seen 25 years of going nowhere. Over these 25 years the composition of households has changed significantly. Single parent families, particularly households headed by single mothers, have grown dramatically as a percent of the population over the past 25 years. That means the median household in the 1980 family was much different from the median household in 2005 – or perhaps better stated, the composition of households below the median changed dramatically over that time and changed in a way that would reduce household income.
Roberts concludes

There has been an increase in workers and households well above the rate of population growth, and that increase is concentrated among below-median earners. That means that if you looked at the median income worker in 1980, that person could have made great progress over time but it is obscured and distorted when you look at the two medians over time.

Roberts points out that if you follow the same type of person over time you will find that their average income went up significantly over the past 25 years. This seems to be a much better use of the data.

I do not have a good feeling on whether income disparity is a larger problem now than in the past but I do believe that the evidence is strongly on the side of those who claim that we are all better off now than we were 25 years ago (or 50 or 100, you name it). Doom mongers should lighten up.