Yesterday, I read an interesting exchange on Twitter between a college student and a college professor about systemic racism, specifically in policing. The college student insisted that in 2018, there was no statistical evidence for the existence of systemic racism in policing. While they were both polite in their debate, I was struck by the student’s unshakable confidence in the objectivity and completeness of statistics. When I was in graduate school studying epidemiology, which is all about biometrics and statistics, my professors routinely encouraged us to maintain a healthy skepticism of all statistical figures and analysis. Unfortunately, I see statistics pop up all the time in discussions about social phenomena, both to prove and disprove their existence or to make very unyielding points. So, I thought it might be helpful to outline and discuss six of the major limitations of statistics. You can find many versions of this list if you do an internet search on “limitations of statistics,” but I’ve adapted what’s available to this post for easier reading (because some are too technical and others are just poorly written).
1. Qualitative aspects are excluded.
Statistical methods cannot study phenomena that cannot be expressed, captured, or measured in quantitative terms. These include capturing the perspectives and lived experiences of certain people or groups of people and dynamic realities such as the emotional impact of racial discrimination. There is a certain danger to relying on only quantitative data and ignoring qualitative data. Writer and historian Hilaire Belloc captured this thought when he wrote, “Statistics are the triumph of the quantitative method, and the quantitative method is the victory of sterility and death.”
Although there has been a heavy bias toward quantitative methods and reporting in published literature, many social science researchers have come to realize the severe limitations of depending primarily on quantitative methodology for understanding human behavior and patterns. For a basic understanding of the major differences between qualitative and quantitative research, see this article.
“It must not be assumed that statistics is the only method to use in research, neither should this method be considered the best attack for the problem.” – Frederick Croxten and Dudley Cowden
2. It does not deal with individual items:
Economist and statistician Horace Secrist explained, “By statistics we mean aggregates of facts…. and placed in relation to each other.” Statistics by nature does not, cannot, and will never be able to take into account individual cases that aren’t placed in relation to any other item. It renders them invisible by subjecting them to the tyranny of aggregates, even though individual cases may be extremely important, informative, and impactful.
3. It does not depict the entire story of a phenomenon:
Most phenomena are affected by a number of factors, not all of which can be expressed quantitatively. Therefore, it’s not possible to examine a problem statistically in all its manifestations and arrive at the correct conclusions. Many phenomena need to be examined in the light of many things—e.g., conditions of life, education, culture, religion, philosophy, administration, history, and other factors that cannot be studied statistically. Thus, it’s important to be mindful that particular aspects expressed numerically do not reliably capture the whole reality of a phenomenon, and any conclusions drawn from such numbers by default exclude and procedurally ignore vital variables, particularly dynamic ones, that directly impact a phenomenon. Bottom line, statistics does not reveal the entire story.
4. It is liable to be misused:
Statistics deals with numbers/figures, which can easily be misinterpreted by untrained people or misused by unscrupulous persons. Either way, it’s very likely to be misused in most cases. According to statistician and economist Wilford I. King, “one of the shortcomings of statistics is that they do not bear on their face the label of their quality.” He also wrote, “Statistics are like clay of which you can make a ‘God’ or a ‘Devil’ as you please,” and also, “the science of statistics is a useful servant but only of great value to those who understand its proper use.” Benjamin Disraeli famously said (and Mark Twain popularized), “There are three kinds of lies: lies, damned lies, and statistics.” Incidentally, this quote, projected onto the wall via an overhead projector, was the way that my Injury Epidemiology professor in grad school introduced her class for the semester in 1996.
5. Its laws are not exact:
There are two fundamental laws of statics: (i) The law of statistical regularity, and (ii) The law of inertia of large numbers. They are both probabilistic in nature and not deterministic like the laws of the physical and natural sciences. Therefore, the conclusions based on these laws won’t be exact like those that are based on the laws of physics or chemistry. When relying on statistical analysis, we can talk only in terms of approximation and probability but not in terms of certainty. For example, according to the law of statistics, if we roll a die 60 times, we are likely to get the number 3 ten times, but this isn’t a guarantee.
6. Its results are true only on an average:
“Statistics largely deals with averages and these averages may be made up of individual items radically different from each other.” —Wilford I. King
Statistical results are not absolutely true and are not applicable to individual cases, even though they’re derived as the average result of all the individuals forming the group. If the average grade of two sections of students is same, it doesn’t mean that all the students in section A got the same marks as all the students in section B. There may actually be large variations between the two. If there are large enough numbers, the numbers can be further clarified by adjusting for certain variables, but in the end, the statistical result is still only useful for general appraisal, done cautiously rather than definitively, and not for any specific unit or event.
Content above adapted from the following references:
- http://www.economicsdiscussion.net/statistics/8-main-limitations-of-statistics-explained/2321
- https://homework1.com/statistics-homework-help/limitations-of-statistics/
Categories: Uncategorized
Life Reconsidered - Judy Wu Dominick 