NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine - https://archive.org/web/.
Read Original Story

Unbalanced Times column reflects coffee confirmation bias

More Consensus on Coffee’s Benefits Than You Might Think

Our Review Summary

Clearly the health effects of coffee is a topic of huge public interest given that coffee is central to our culture, our morning routine, and our social lives — and likely one of the most commonly consumed beverages on the planet. This report includes links to numerous meta-analyses about the effects of coffee on a wide range of health impacts including heart disease, cancer, diabetes, liver disease, Alzheimer’s and Parkinson’s disease. The underlying assumption is that coffee has a bad rap that is not deserved, given the vast amount of evidence that has accumulated about its potential benefits over the years. One might be tempted to pour an additional cup every morning if, as the article suggests: “the potential health benefits are surprisingly large.”

While this is a compelling and intriguing report that reflects considerable research by the author, one can’t help but think it suffers from a bit of confirmation bias which is described as: “the tendency to search for, interpret, or recall information in a way that confirms one’s beliefs or hypotheses.” Seems that when the Times went out to look for research on the effects of coffee — both the good and the bad — the bad wasn’t generally found. What would have helped was some sense of bias that might creep into the available coffee-related research, that it is mostly observational, and hence subject to a variety of limitations that are less likely found in randomized studies. Nearly every study of coffee consumption relies on peoples’ perceptions and recall of how much coffee they thought they drank, and we know that recall bias would affect all of those studies where self-reported behaviors are being measured. What the reader needs, up front, in an article such as this is a more rigorous probing of the common confounders in observational research, the potential for bias (including publication bias, reporting bias, social responsibility bias, and others) and so on. We suspect that there may even be healthy user bias and the possibility, even remote, that the people who respond to surveys about how much coffee they drink are not representative of the general population and hence the study results are less likely to be seen in the “real world.”

 

Why This Matters

Most of us consume coffee, regularly, so most of us would be somewhat interested in knowing if coffee is likely to improve or degrade our health. Media reporting of research on nutrition habits is very squidgy territory and consumers often feel like the media is swinging them on a pendulum from the “coffee is evil” stories to the “coffee is the elixir of life” stories. Since we often consume media while we consume coffee, we should be reminded of the very human tendency to devour the stories that confirm our biases and ignore those that don’t. Regardless of how impressed we are by, and how much we want to believe in the vast body of coffee consumption research, we shouldn’t generally use it as a gold standard to guide our decision making. Our fear in a story like this is there are many, many people who really dislike coffee, and we worry that the apparent preponderance of evidence reported here will make them think that they need to be choking back a couple of cups a day “just in case.”

Criteria

Does the story adequately discuss the costs of the intervention?

Not Satisfactory

There is no discussion of costs, but we’ll rate this Not Applicable since most coffee consumers would have some idea of how much their habit costs them.

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The range of studies cited and their conclusions is impressive, but the actual quantification of benefits is scant and sometimes misleading. Look at the quantification of benefits related to liver cancer which gives a relative risk reduction. “A meta-analysis published in 2007 found that increasing coffee consumption by two cups a day was associated with a lower relative risk of liver cancer by more than 40 percent.” Really? 40% of what? My lifetime risk of liver cancer to start with, according to the American Cancer Society, is “1 in 81, while an average woman’s risk is about 1 in 196”  (this equates to 1.23% for men and 0.51% in women). Reducing my (a man’s) relative risk of liver cancer then goes from 1.2% down to 0.72%, a difference of about 0.48%.  Is that worth an extra 2 cups of coffee per day? If you don’t like the stuff, again, you need to know these slight differences.

Does the story adequately explain/quantify the harms of the intervention?

Satisfactory

The story warns that adding sugar and fat to coffee may negate any potential health benefits. The story also mentions “jitteriness” and the fact pregnant women should limit consumption, but doesn’t describe the potential harms of excess caffeine in pregnancy that have been well described in the literature. And there are a number of studies suggesting that caffeine seems to trigger symptoms in people with anxiety disorders. More generally, what about the research on the ill effects of coffee on insomnia, or urinary incontinence. or dependence?  What about withdrawal headaches and the problems of “stopping” coffee drinking?  We’ll give the benefit of the doubt here as we usually do on borderline calls, but the exploration of harms was not nearly as thorough as the coverage of benefits.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

Using meta-analyses is very good reporting, and clearly the story deserves credit for examining the research on coffee in considerable depth. And yet such reporting must come with important caveats such as sufficient detail on the very real biases at play and the other limitations of observational studies. We have to wade 18 paragraphs into this story, through a giant mug full of studies showing benefits of coffee on every health condition imaginable, before we receive a weak restraining comment about the lack of randomized trials in this area. That’s too little, too late.

Does the story commit disease-mongering?

Satisfactory

There is little evidence of disease mongering here, and it’s not really relevant as coffee is not generally drunk because of any diseases whose prevalence is exaggerated.

Does the story use independent sources and identify conflicts of interest?

Not Satisfactory

An independent expert would have added valuable context here, but no such voice is provided.

Are studies of coffee consumption which find themselves as part of meta-analyses of coffee consumption funded by, tied to, or otherwise linked to Big Coffee in any way that suggests commercial biases in this research?  We don’t know, and we don’t learn about any such conflicts of interest in the story. Is it a conflict of interest if the reporter, plus most of the researchers and meta-analyzers are routine, habitual coffee consumers?

Does the story compare the new approach with existing alternatives?

Not Applicable

Not really applicable in this setting, though tea research might have been used as a comparison.

Does the story establish the availability of the treatment/test/product/procedure?

Satisfactory

Coffee is everywhere, in every culture, and in every corner of the planet — and the story mentions places you can get it.

There is an intersection in Vancouver with a Starbucks on EVERY corner.

Have you heard about the new Starbucks that opened in the bathroom of an existing Starbucks?

Does the story establish the true novelty of the approach?

Satisfactory

The story does not claim that research showing potential benefits of coffee is new, and it describes the publication dates of numerous studies.

We’ll rate the story Satisfactory on that basis. However, the news hook for the story is that it’s pointing out research that has been overlooked — that it’s helping to change the public’s perception of coffee from vice to something more virtuous. We think that premise is dubious at best. Given the countless thousands of articles about the benefits of coffee in recent years, is it really news to anyone that coffee’s image may deserve a makeover?

Does the story appear to rely solely or largely on a news release?

Satisfactory

It’s difficult to envision a news release that could account for this story.

Total Score: 5 of 9 Satisfactory

Comments

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.