VIDEO: Does diet soda = dementia? A second look at the study uncovers key statistical shortcomings

Just over a week ago I wrote about how reporters covering a recent observational study — which reported that drinking at least one artificially sweetened beverage daily was associated with an increased risk of developing stroke or dementia — often neglected three key points.

Those points were:

• relative risk was reported much more often than absolute risk
• that association does not equal causality was not always mentioned
• key limitations of the study were rarely included.

But there was one key limitation I did not include that one of our frequent contributors — Dr. Vinay Prasad — jumped on right away. He said that given the huge number of variables in the study (over 90), the authors should have performed what’s known as a “multiple comparisons” adjustment. Without such an adjustment, it’s likely that some results would turn up positive simply due to the play of chance.

This limitation was mentioned by the authors at the very end of the paper:

“… we did not adjust for multiple comparisons meaning that some findings may be attributable to chance”

But it didn’t find its way into any of the news coverage about the study that I saw. 

This is significant because — as you can imagine — any study suggesting an association between drinking diet soda and the risk of dementia and stroke, is going to get massive coverage worldwide.

Which this study did.

Maybe that’s why when Dr. Prasad tweeted his indignation, many in the science community took note, and felt strongly about retweeting such an important drawback of the study.

Perhaps you are like me in that your statistics muscle doesn’t get flexed much. So I needed help to better understand what got Dr. Prasad (and his Twitter followers) all worked up. So I turned to one of our regular contributors,  Dr. Susan Wei , a mathematician and statistician at the University of Minnesota School of Public Health.

Most of us don’t have a natural aptitude for statistics. And I’m not saying we need to run out and take a crash course in it just so we can scrutinize the news.

But here’s something I learned from covering this study. As always, look closely at the limitations of the study. Most authors include them somewhere. And if there’s a single limitation which is unclear, find a source who can translate it for you in terms you and your readers can understand.

In this case social media proved invaluable. Although I wouldn’t consider Twitter a vetted source, it can certainly direct you to some knowledgeable experts.