This WebMD story repeatedly uses language that suggests an active, causal link between chocolate and stroke (e.g. reduce, protect, benefit), when the study — as with all observational studies — was not capable of determining whether chocolate in fact prevents strokes. We provide a primer for journalists and others about how to describe observational studies; reading it could help prevent these problems. Other shortcomings include the use of relative rather than absolute risk comparisons, and lack of a clear description of potential harms. These problems overshadow the story’s bright spots, including the perspective of an independent expert and caveats about the differences between U.S. and Swedish chocolate.
Overstating the results of observational studies is a big problem in health journalism, and it has serious consequences. Since the conclusions of observational studies are frequently overturned by more rigorous research, hyping the results of these early studies can contribute to the sense that researchers are always “flip-flopping” on their recommendations and that they essentially don’t know what they’re talking about. In reality, researchers are usually pretty careful to explain what their studies can and can’t tell us; journalists have to get better at communicating those caveats to readers.
The story does a good job of emphasizing that health benefits are more likely with chocolate that has a higher concentration of cocoa solids. However, these premium chocolates come with a corresponding increase in price compared with your average Hershey Bar– something the story could have explained. Nonetheless, we’ll rule this not applicable as most people probably have a general idea about the cost of chocolate.
The story says that researchers observed a 20% reduction in strokes among participants eating the most chocolate relative to those eating the least — a statistic that conveys very little useful information. Compare this with the much more informative Reuters description that provides absolute risks from data contained in one of the study tables: “Among those with the highest weekly chocolate intake, more than 45 grams, there were 2.5 strokes per 1,000 women per year. That figure was 7.8 per 1,000 among women who at the least, less than 8.9 grams a week.”
Although the story warns that milk chocolate may be “high in sugar, fat, and calories,” it doesn’t spell out the implications of eating too much milk chocolate in an attempt to reap health benefits. (To the contrary, the story suggests that eating larger amounts of U.S. milk chocolate with low cocoa content might yield the same result as eating less of the higher-cocoa Swedish chocolate.) Eating more than a modest amount of chocolate is likely to cause weight gain, which would probably negate any beneficial effects on stroke and increase the risk for other health problems.
Starting with the headline, which claims that “Chocolate May Cut Women’s Stroke Risk,” the story consistently misstates this observational study’s conclusions regarding the relationship between chocolate consumption and stroke. Instead of reporting the study’s finding of an association between chocolate and stroke risk, the story repeatedly suggests that chocolate was responsible for the lower rate of strokes seen in those who ate more chocolate. For example: “The group eating the most chocolate got the most benefit, reducing stroke intake [sic] by 20%.” For comparison, see how the competing Reuters story handled this key limitation by soliciting a quote from one of the study authors, who said: “Given the observational design of the study, findings of this study cannot prove that it’s chocolate that lowers the risk of stroke.”
The story did do a nice job of pointing out that milk chocolate in Sweden, where the study was conducted, is richer in cocoa solids than milk chocolate in the U.S., and so a study conducted here might not report the same findings on stroke risk.
The story didn’t exaggerate the effects of strokes.
The study includes comments from an expert source not affiliated with the study, who provides valuable context on the increased desirability of dark chocolate over milk chocolate.
The story could have included a line about other, more established methods of preventing cardiovascular disease, but didn’t.
The availability of chocolate bars is not in question. But it is important to note that the chocolate sold in Sweden typically has a higher concentration of cocoa solids — the likely source of any protective effect — than chocolate sold in the U.S. The story makes this clear.
The story mentions other research suggesting that chocolate consumption is linked to reduced risk of strokes.
Because the study includes comments from an independent expert, we can be sure it didn’t rely excessively on any press release.