Coming to the Wrong Conclusion: A Case Study of How Misinterpreting Data Obscures the Facts

Two English newspaper articles from 2011 compared with an earlier quantitative study highlight the problem of drawing inferences from the statistics of social sciences research. Both articles use statistics to bolster their identical yet flawed claims. The study shows the error in their reasoning. Examining the articles’ use of statistics shows just how easy it is to use perfectly good data to draw unfounded conclusions. And, it reminds us that when we hear a claim backed up with statistics, we must ask what those statistics really mean.

The Guardian (Batty, 2011) reports on a perceived increase in depression in England. The article concludes with a quote from the chief executive of Depression Alliance that summarizes what the writer wants us to believe about these statistics: “These uncertain economic times are linked to an increase in the number of people with the illness.” The article alternates between quantitative statistics and cautionary quotes that play on emotions. For example, it follows “People who are unable to work due to depression lose £8.97bn of potential earnings a year, while the loss of earnings from suicide is put at £1.47bn,” with a quote from the commissioner of the studies “Failure to tackle depression hurts us all. It makes a misery of the lives of sufferers…” The statistics include the documented increase in number of antidepressant prescriptions, sourced from a study by the NHS. (NHS is England’s National Health Service, a publicly funded healthcare system.) However, reliable as the data may be from NHS regarding increased numbers of prescriptions, nowhere does the article question whether another factor comes into play.

The Independent (Morris, 2011) uses the exact same NHS statistics as The Guardian in their reporting. They also go into more depth with additional statements such as, “The breakdown of the statistics suggested that women were nearly four times more likely than men to approach their doctor with the problem.” The article further cites increases in sleeping pills and anti-anxiety medication. But, all of these stats simply lead The Independent to the same conclusion as The Guardian: depression is on the rise in England.

Dr. Ben Goldacre (2011) reveals the main assumption in these articles: “They identify that the number of individual prescriptions written for antidepressant drugs has risen, and then assume this means that more people are depressed.” The NHS study has given a statistic and the newspapers have drawn their conclusion. But might there be another cause of the increase? In fact, such a factor has been previously documented.

As Goldacre points out, these two articles would benefit from the more in-depth analysis published two years prior in BMJ (formerly British Medical Journal.) The Independent and The Guardian only use the increasing number of prescriptions to draw their conclusions. The study published in BMJ, however, goes much farther than this simple figure. It digs beyond the surface statistic to analyze 189,851 individual patients’ prescription histories. They did not find an increase in depression cases, but an increase in the length of time patients were treated with prescription anti-depressants. The total number of prescriptions did not translate into more cases at all (Moore, 2009). Looking into the study shows how one could see an increase in the prescription rate yet also see a decline in the overall number of cases. It provides precisely the level of detail, analysis, and critical thinking that the newspaper articles lack.

The BMJ study covers the years 1993 to 2005. They found that the number of prescriptions per patient “increased from 2.8 in 1993 to 5.6 in 2004.” But in terms of the number of patients, both men and women showed noticeable decline: from 1993 to 2005, the rate per 1000 patients dropped in men from 7.83 to 5.97 and in women dropped from 15.83 to 10.06. In terms of their depth of data analysis, the BMJ study also includes comprehensive graphics and a full report on their methods. The newspaper articles boil down to “the NHS said this and here is what it means.” But, the formal study actually lays out the data in clear and specific form so that readers can see how they arrived at their conclusions. As Goldacre summarized the findings of the BMJ report,

“The rise in the overall number of antidepressant prescriptions was not due to increasing numbers of patients receiving antidepressants. It was almost entirely caused by one thing: a small increase in the small proportion of those patients who received treatment for longer periods of time.”

Goldacre makes no claim that the exact same cause explains the 2011 trend reported in The Independent and The Guardian. However, he reminds us that before we make claims about the meaning of our highly-touted “statistical facts,” we need to look more closely into other potential causes for trends. Flaunting descriptive statistics makes one sound knowledgeable and ‘scientific,’ but the true scientist and analyst knows to dig deeper into the data. Often the ‘facts’ reported by the news media can hide the truth rather than reveal it. It is up to the social science researcher to reach beyond the allure of the face value and push for deeper analysis of the facts.

References

Batty, D. (2011, December 29). Antidepressant use in England soars. The Guardian. Retrieved from http://www.guardian.co.uk/society/2011/dec/30/antidepressant-use-england-soars

Goldacre, B. (2011, December 30). These Guardian/Independent stories are dodgy. Traps in data journalism. Bad Science. Retrieved from http://www.badscience.net/2011/12/this-guardian-story-is-dodgy-traps-in-data-journalism

Moore, M. (2009). Explaining the rise in antidepressant prescribing: a descriptive study using the general practice research database. BMJ 2009, 339:b3999. Full text retrieved from http://www.bmj.com/content/339/bmj.b3999

Morris, N. (2011, December 30). Escalating depression crisis is costing Britain £11bn a year. The Independent. Retrieved from http://www.independent.co.uk/life-style/health-and-families/health-news/escalating-depression-crisis-is-costing-britain-11bn-a-year-6282994.html

Add Your Insight

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s