Michael Joyce produces multimedia at HealthNewsReview.org and tweets as @mlmjoyce
Just over a week ago I wrote about how reporters covering a recent observational study — which reported that drinking at least one artificially sweetened beverage daily was associated with an increased risk of developing stroke or dementia — often neglected three key points.
Those points were:
But there was one key limitation I did not include that one of our frequent contributors — Dr. Vinay Prasad — jumped on right away. He said that given the huge number of variables in the study (over 90), the authors should have performed what’s known as a “multiple comparisons” adjustment. Without such an adjustment, it’s likely that some results would turn up positive simply due to the play of chance.
This limitation was mentioned by the authors at the very end of the paper:
“… we did not adjust for multiple comparisons meaning that some findings may be attributable to chance”
But it didn’t find its way into any of the news coverage about the study that I saw.
This is significant because — as you can imagine — any study suggesting an association between drinking diet soda and the risk of dementia and stroke, is going to get massive coverage worldwide.
Maybe that’s why when Dr. Prasad tweeted his indignation, many in the science community took note, and felt strongly about retweeting such an important drawback of the study.
Perhaps you are like me in that your statistics muscle doesn’t get flexed much. So I needed help to better understand what got Dr. Prasad (and his Twitter followers) all worked up. So I turned to one of our regular contributors, Dr. Susan Wei , a mathematician and statistician at the University of Minnesota School of Public Health.
Most of us don’t have a natural aptitude for statistics. And I’m not saying we need to run out and take a crash course in it just so we can scrutinize the news.
But here’s something I learned from covering this study. As always, look closely at the limitations of the study. Most authors include them somewhere. And if there’s a single limitation which is unclear, find a source who can translate it for you in terms you and your readers can understand.
In this case social media proved invaluable. Although I wouldn’t consider Twitter a vetted source, it can certainly direct you to some knowledgeable experts.
Comments (4)
Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.
Paul Alper
May 3, 2017 at 5:00 pmOnce again, Michael Joyce has written a terrific article. However, his comment,
“Most of us don’t have a natural aptitude for statistics. And I’m not saying we need to run out and take a crash course in it just so we can scrutinize the news”
is disheartening to this former statistics teacher. As long as society keeps relegating the subject of statistics to the dull and inscrutable aspects of existence, we will remain prone to misunderstanding quantitative results. Unlike, for example, the difference between relative risk and absolute risk, the dilemma of multiple comparisons is always discussed in an elementary statistics course, even if how best to handle it is complicated.
Andrew DePristo
May 5, 2017 at 7:27 amI like the simple statistical based argument by Dr. Prasad but such an argument also makes it appear that the only problem with the study is statistical treatment. In my original criticism, I pointed out that the treatment of a major confounding variable, blood pressure, could not be treated properly because of its high hazard ratio, 5 to >30, depending upon exactly how high the BP is (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3838588/). The correction for this would have to be done for each individual as a function of time since BP varies over the duration of the study. I don’t believe that anyone knows how to do such a correction. As such, even if the findings reached statistical significance according to Dr. Prasad’s p<0.0005 value, the correction methods would still be flawed and the evidence incorrect. It is the science that is flawed not just the statistical analysis!
Nina Teicholz
May 8, 2017 at 5:38 amExcellent job! I would love to see this kind of analysis for the many epidemiological studies that are published. Keep it up!
Doug Mathias
May 8, 2017 at 9:55 amI agree with Paul Alper’s comment that statistics, studied formally (and probably on a “project” basis rather than a “hard theory” basis in my opinion) should be part of every adult’s ongoing educational curriculum.
I am a bit confused as to why elevated BP should be treated entirely as a confounding variable however. This is not my field, but effective research design must be the key to exposing causality here, and I suspect it is less “the science” than the design. Can anyone enlighten me on this?
Our Comments Policy
But before leaving a comment, please review these notes about our policy.
You are responsible for any comments you leave on this site.
This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.
We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.
We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.
There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”
The authors of the Retraction Watch comments policy urge commenters:
We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.
And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.
The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.
You might also like