Kathlyn Stone is an associate editor at HealthNewsReview.org. She tweets as @KatKStone.
A couple of recent headlines dealing with complementary and alternative medicine (CAM) caught our attention:
When it sounds too good to be true, it probably is. When we reviewed these stories, we found that in both cases journalists were not adequately scrutinizing the studies and ultimately mischaracterized the findings.
In our review of the HealthDay story, we disagreed with the premise of the article, which is evident in our headline, “Acupuncture in the ER: No, study did not prove it was ‘safe and effective.’”
That’s because the published study determined that “none of the examined therapies provided optimal acute analgesia.”
Meanwhile, the CBS News story was about a study that compared the benefits of three different interventions – yoga, physical therapy and reading educational materials – on alleviating chronic back pain.
Our takeaway: While the CBS News story made it clear that yoga wasn’t a clear winner, it left out a key detail: Neither yoga nor physical therapy were statistically better than education (the control group).
In both story examples, it appears as though journalists weren’t examining the source studies critically and perhaps relying on a PR release that put a positive spin on the findings.
So what is going on? Why are non-conclusive CAM studies being framed so positively, with no or few cautions about the research?
Tim Caulfield has some ideas. He is one of North America’s leading skeptics on popular health advice, particularly that of the celebrity variety. Caulfield is research director of the Health Law Institute at the University of Alberta. But he’s probably better known among health news analysts as the author of the book Is Gwyneth Paltrow Wrong About Everything?
“I suspect there is a good deal of ‘white hat’ bias in the coverage of CAM studies,” Caulfield told me in an email. White hat bias refers to a bias toward research that’s not funded by industry and may involve “cherry-picking” the evidence to make it seem more significant than it is.
“For example, yoga is a light exercise that is popular and pretty noninvasive. Who wouldn’t want it to work as a therapy? But the media needs to take greater care when evaluating these studies,” Caulfield said. “What is the intervention being compared to? Often, the media reports a positive result when the results could equally be described thus: ‘this CAM intervention works as well as other stuff that doesn’t work well.'”
Caulfield also pointed out that it is often near impossible to properly blind these studies, so the placebo effect and “regression to the mean” (or an averaging out of the results over time) almost always play a big role.
“Finally, I think people assume that there is no science hype in the context of CAM. But, as studies have demonstrated, there is often a strong tendency to publish positive results in this context.”
For the yoga/back pain study we mentioned earlier, it wasn’t just CBS News that made yoga sound more effective than it was. These news outlets committed the same mistake:
While there may not be big bucks in alternative therapies–at least not when compared to, say, cancer treatment–journalists still need to scrutinize the findings as closely as they would any other study.
As Caulfield notes, “just because it is CAM research, doesn’t mean hype isn’t a problem.”