We don’t think this story provided answers to women, as it failed on most of our criteria – all of which are intended to address elements of a story that we think consumers need in order to evaluate studies and claims about health care interventions.
We need to help readers evaluate the quality of evidence.
What does it mean when a meta-analysis looks at studies of soy versus placebo?
What might be the limitations of such an approach?
What have we done to make news consumers/health care consumers any smarter by this story?
Not much, we’re afraid.
No discussion of cost.
The story dances all around – but never delivers specific data on the absolute benefit seen in the meta-analysis.
The closest it comes is this: “soy supplements with higher amounts of the isoflavone called genistein were more than twice as good at reducing hot flash frequency than those with low amounts.”
And this non-evidence statement from one of the author-researchers: “What this study shows is that ingesting soy isoflavones will help you. I personally think foods [containing soy] are better.”
No discussion of potential harm.
The study on which the story is based is a meta-analysis, which combines multiple studies that, taken alone, may fail to show any difference between treatments, to see if the pooled results provide greater statistical power to detect a small difference that might be missed in a single study. If done properly, a meta-analysis can provide useful insights. But the statistical methods are complex and the devil is in the details.
At the end, an independent expert briefly raises this point – “Because the studies vary greatly in their methods and the preparations used, we still don’t know who are women who will benefit.”
But by then the story had driven home the suggestion of definitive benefit – in the headline and in ensuing copy – and it was too little too late.
To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better” – but with no data provided to support that opinion.
No disease-mongering, but, then again, there wasn’t much information about hot flashes at all in the story.
Perhaps the only saving grace of the story: One clearly skeptical independent expert and one other who noted limitations.
The story only stated, “Soy has been touted as an alternative treatment to hormone replacement therapy after HRT was linked to an increased risk of breast cancer.”
But there’s still a decision to be made by women – taking both harms and benefits into account. As already noted in the “Benefits” criterion comment above, the story never delivers specific data on the absolute benefit seen in the soy vs. placebo meta-analysis. So it paints HRT as risky, and then promotes soy’s benefits with its definitive headline “Soy supplements can cool hot flashes” and other language in the story – without ever clearly defining the scope of the benefit.
What was the size of the benefit? What was the size of the tradeoff of potential harms? The story didn’t deliver that information – so a meaningful discussion of alternatives was not provided. One could imagine a very helpful infographic being created to explain this.
To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better.”
The availability of soy supplements is not in question.
The story at least quoted another researcher, whose own work “published in 2011 in the Archives of Internal Medicine, she found no effects on menopausal symptoms when women took 200 mg of soy isoflavones for two years.”
Because two independent experts were quoted, we don’t think there’s any evidence that the story relied on a news release.