Total Posts:13|Showing Posts:1-13
Jump to topic:

The dangers of cynicism towards statistics

Diqiucun_Cunmin
Posts: 2,710
Add as Friend
Challenge to a Debate
Send a Message
9/15/2016 5:11:05 PM
Posted: 2 months ago
I apologise for the rather opaque title. I had to write succinctly because of the limited space. By 'cynicism' I refer to an attitude that is increasingly prevalent, including (especially?) among educated people and DDO users. It is a general distrust towards quantitative evidence, a belief that statistics are too prone to manipulation to be regarded as a reliable source of information, that you can make data to say whatever you want. It does not always amount to seeing ulterior motives behind every figure, which the word 'cynicism' may imply, but is often realised in a way that is probably worse: People see ulterior motives behind statistics incompatible with their worldview. The disbeliever in statistics is just as prone to confirmation bias than the believer.

Like many members of DDO, I am disillusioned about the way numbers and statistics can be bended and twisted to spin nearly any narrative. Quantitative data are not perfect, and rarely show us the whole picture. But in many cases, it is the best source of information we have. Cynicism about the use of statistics hampers our ability to improve the quality of our decisions: Cynics usually turn to personal experience, anecdotal evidence and appeals to authority, and almost always fare worse as a result.

I'm not saying figures are the be-all and the end-all of everything. Personal experience, for example, has its place when it comes to matters like health and parenting. Different body types thrive on slightly different kinds of nourishment, and not all kids have the same temperaments. Sticking to what is statistically the 'best' diet or 'best' parenting method does not always yield optimal results. Yet when it comes to formulating public policy or making generalised statements about the world (science), surely the data is key.

It is understandable why some believe that 'all data can be twisted in any way to prove a point'. What I want to show is, as long as we have access to the data, we *can* do better. We *can* use figures to provide accurate answers to the questions we have. While you can 'say whatever you want' about statistics, the conclusions you draw are usually either right or wrong.

Imagine it's 1987, and somebody wrote this in a newspaper:

Ah, the death penalty in Florida. What if I told you that 4.8% of whites convicted of murder are executed, but only 2.5% are not?

Is it time to start declaring court bias against whites? Well, the first thing to notice is that these statistics were taken from an observational study. The researchers didn't pick citizens at random, order them to murder someone and have them tried. Most observational studies involve one important risk: Confounding factors. When we are looking for a relationship between an independent variable (defendant's race) and a dependent variable (death penalty or not), it could be that there is a hidden variable. In this case, the hidden variable is the victim's race.

Now look at the new data:
Black-on-black: 0.8% executed
White-on-black: 3.4% executed
Black-on-white: 12.6% executed
White-on-white: 4.9% executed

It turns out that when you look at the victim's race as well, a new pattern appears. When the victim is black, the murderer is far less likely to be executed. So, rather than implicating an anti-white bias, the courts seem to have demonstrated a bias against black victims. We can explain the newspaper's statistic by noting that whites are more likely to kill whites and blacks more likely to kill blacks.

But that's not the end of the story. There are other factors to take into account. What if we can separate the data into even finer details? There are other possible factors. It could be that white victims are more likely to be killed in mass murders. After all, whites make up a larger proportion of the population, so when there are mass murderers, they're more likely to kill whites than blacks. That sounds reasonable, so we see if this is a relevant factor.

If we only look at multiple-murderers, 11.0% of white defendents and 7.9% of black defendents are executed. But when the victim is white, 11.3% of whites and 22.9% of blacks are executed; when the victim is black, no whites and 2.8% of blacks are executed. (This reversal of the relationship is very common and known as Simpson's paradox, for those who are interested.) This buttresses our hypothesis that murderers with white victims are more likely to be executed than those with black victims, and further reveals a possible court bias against black defendents as well.

I won't go into all the possible confounding factors, but suffice it to note that the Radelet and Pierce (1991) study, a rather classical case in categorical data analysis, concluded that none of the factors they looked at could eliminate the racial bias on victims (while the racial bias against black defendents could be explained away). This implies that there is likely to be a tendency for Florida courts not to execute murderers who killed blacks.

A rightist could have looked at the 4.8%/2.5% disparity and declared reverse discrimination against whites defendants. A leftist could have looked at the figures for multiple murders, and declared discrimination against black defendants. But there is a correct conclusion - that murderers of black victims are less likely to be executed, though murderers' race don't affect rulings - that we can draw from the data. Statistical cynics are incapable of noticing this nuance.

So what can we do about statistical cynicism? The best way is to change the default mindset that we ourselves adopt when we are faced with statistics. Here are a few pointers:

-Ask the right question. This wasn't really covered in my example, but asking the wrong question can lead to wrong conclusions. Consider a hypothetical scenario in which we wanted to know how to reduce crime. If we look at the statistics and find that a immigrants have a higher per capita rate of committing crime, this doesn't imply that we need to halt immigration, because immigrants could make up a negligible percentage of the population. After all, it's the overall crime rate that we want to reduce. By contrast, let's imagine we are instead doing a scientific study on whether immigrants are more likely to be criminals. In this case, the fact that immigrants have a higher per capita crime rate can establish the veracity of this hypothesis.

-Look at methodology. In the above, I noted that the study was an observational study, hence the emphasis on confounding variables. However, this study is a census in the statistical sense. If it were a sample study, we would have to look at source of possible sampling bias, such as non-response bias.

-Familiarise yourself with basic statistical theory. It's understandable that you're not too fond of proving that the convolution of a series of exponential random variables is a gamma random variable, but the intuitive meaning behind concepts like joint, condition and marginal distributions should pose no difficulty. Similarly, while you may not be acquainted with the mathematical details of the Cramer-Rao Lower Bound in point estimation, you can at least understand how one-sample and two-sample t-tests work. Basic statistics can get you quite far in most cases.

The quantitative method is one of greatest inventions of modern intellectual history, because of its robustness, precision and breadth of applicability. In the era of big data, where everything and anything can be measured and quantified, the importance of numbers will only increase by the day. If you still harbour a cynical attitude towards numbers, please replace it with an attitude of healthy scepticism and embrace the wave of change that the Information Age has unleashed.

References:
Radelet, M. L., & Pierce, G. L. (1991). Choosing those who will die: Race and the death penalty in Florida. Florida Law Rev., 43, 1-34.
The thing is, I hate relativism. I hate relativism more than I hate everything else, excepting, maybe, fibreglass powerboats... What it overlooks, to put it briefly and crudely, is the fixed structure of human nature. - Jerry Fodor

Don't be a stat cynic:
http://www.debate.org...

Response to conservative views on deforestation:
http://www.debate.org...

Topics I'd like to debate (not debating ATM): http://tinyurl.com...
Diqiucun_Cunmin
Posts: 2,710
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 6:46:09 AM
Posted: 2 months ago
At 9/16/2016 5:32:16 AM, Beisht_Kione wrote:
Excellent essay.

Thanks :)
The thing is, I hate relativism. I hate relativism more than I hate everything else, excepting, maybe, fibreglass powerboats... What it overlooks, to put it briefly and crudely, is the fixed structure of human nature. - Jerry Fodor

Don't be a stat cynic:
http://www.debate.org...

Response to conservative views on deforestation:
http://www.debate.org...

Topics I'd like to debate (not debating ATM): http://tinyurl.com...
keithprosser
Posts: 2,061
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 7:36:47 AM
Posted: 2 months ago
It seems to me that most of the time all we get are bald numbers without enough context to judge them. In that case cynicism would seem only prudent.
keithprosser
Posts: 2,061
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 12:49:15 PM
Posted: 2 months ago
I would say I rarely question or deny statistics. I am perfectly happy to accept blacks score lower in IQ tests for example. The cynicism comes in accepting the interpretation of those statistics some people promote.
dylancatlow
Posts: 12,254
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 2:03:23 PM
Posted: 2 months ago
In your "Objectivity in science" thread, I may have given the impression that I'm cynical toward statistics. I'm not really. I think they are very useful and often the best tool we have for formulating policy, and personally I love looking at them. But they can be and often are misused by people who don't really understand them.
Skepsikyma
Posts: 8,286
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 2:59:40 PM
Posted: 2 months ago
I have the opposite impression about public sentiment, though perhaps it can be chalked up to societal differences. I think that people in the US, especially those who perceive themselves as exceptionally intelligent, are credulous to the point of absurdity when it comes to statistics, though there is a distinct minority which is incredulous to the point of absurdity. Mostly I notice that a perfectly valid non-statistical argument will be ignored as unserious. I trace this to the US education system, which puts much emphasis on amorphously defined 'evidence' and very little on logic. I also think that statistics have a broad limit to their application which is often ignored, and past that limit people just use manipulations of proportion to basically say anything they want (psychology, parenting, and nutrition are excellent examples, and you used them to make a similar point).
"The Collectivist experiment is thoroughly suited (in appearance at least) to the Capitalist society which it proposes to replace. It works with the existing machinery of Capitalism, talks and thinks in the existing terms of Capitalism, appeals to just those appetites which Capitalism has aroused, and ridicules as fantastic and unheard-of just those things in society the memory of which Capitalism has killed among men wherever the blight of it has spread."
- Hilaire Belloc -
kevin24018
Posts: 1,952
Add as Friend
Challenge to a Debate
Send a Message
9/16/2016 7:12:49 PM
Posted: 2 months ago
there's no danger in it, if you look at them factually and for what they are, most I have seen are done by people trying to prove their point or agenda. The context is rarely accurate or easily discernible. Not to say there aren't accurate ones, but we should scrutinize them all to keep them honest.
Diqiucun_Cunmin
Posts: 2,710
Add as Friend
Challenge to a Debate
Send a Message
9/19/2016 4:21:33 AM
Posted: 2 months ago
At 9/16/2016 7:36:47 AM, keithprosser wrote:
It seems to me that most of the time all we get are bald numbers without enough context to judge them. In that case cynicism would seem only prudent.

I would call that scepticism rather than cynicism - there are people who outright deny the value of statistics. Though most of the time, it is possible to find the sources of the numbers we see in the media - they should cite where their sources come from, even if it's a simple 'figures from ___ show'. If not then, honestly, it may be the time to get a better source of information.
The thing is, I hate relativism. I hate relativism more than I hate everything else, excepting, maybe, fibreglass powerboats... What it overlooks, to put it briefly and crudely, is the fixed structure of human nature. - Jerry Fodor

Don't be a stat cynic:
http://www.debate.org...

Response to conservative views on deforestation:
http://www.debate.org...

Topics I'd like to debate (not debating ATM): http://tinyurl.com...
missmozart
Posts: 306
Add as Friend
Challenge to a Debate
Send a Message
9/19/2016 4:17:37 PM
Posted: 2 months ago
At 9/16/2016 5:32:16 AM, Beisht_Kione wrote:
Excellent essay.
"Bonjour" -Feu

Diqiu: "Asian men are generally perceived as more feminine..."
Me: "Are you feminine?"
Diqiu: "Hey, no!"

"Do really really really good pens turn you on?" -Hayd

"bsh1's profile pic is what the snapchat filter would look like on steroids"- VOT

"let's keep it simple and traditional :D" -Biodome
Axon85
Posts: 137
Add as Friend
Challenge to a Debate
Send a Message
9/19/2016 5:31:20 PM
Posted: 2 months ago
I think people love stats... so long as those stats corroborate their beliefs. Cynicism often emerges when a statistic or scientific discovery challenges one's ideology or cherished assumptions.
keithprosser
Posts: 2,061
Add as Friend
Challenge to a Debate
Send a Message
9/21/2016 2:16:20 PM
Posted: 2 months ago
At 9/19/2016 5:31:20 PM, Axon85 wrote:
I think people love stats... so long as those stats corroborate their beliefs. Cynicism often emerges when a statistic or scientific discovery challenges one's ideology or cherished assumptions.

What does 'cynical' actually mean, anyway?