Soy Supplements Can Cool Hot Flashes: Study

Does the story adequately discuss the costs of the intervention?

Not Satisfactory

No discussion of cost.

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The story dances all around – but never delivers specific data on the absolute benefit seen in the meta-analysis.

The closest it comes is this:  “soy supplements with higher amounts of the isoflavone called genistein were more than twice as good at reducing hot flash frequency than those with low amounts.”

And this non-evidence statement from one of the author-researchers: “What this study shows is that ingesting soy isoflavones will help you. I personally think foods [containing soy] are better.”

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

No discussion of potential harm.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

The study on which the story is based is a meta-analysis, which combines multiple studies that, taken alone, may fail to show any difference between treatments, to see if the pooled results provide greater statistical power to detect a small difference that might be missed in a single study.  If done properly, a meta-analysis can provide useful insights.  But the statistical methods are complex and the devil is in the details.

At the end, an independent expert briefly raises this point – “Because the studies vary greatly in their methods and the preparations used, we still don’t know who are women who will benefit.”

But by then the story had driven home the suggestion of definitive benefit – in the headline and in ensuing copy – and it was too little too late.

To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better” – but with no data provided to support that opinion.

Does the story compare the new approach with existing alternatives?

Not Satisfactory

The story only stated, “Soy has been touted as an alternative treatment to hormone replacement therapy after HRT was linked to an increased risk of breast cancer.”

But there’s still a decision to be made by women – taking both harms and benefits into account.  As already noted in the “Benefits” criterion comment above, the story never delivers specific data on the absolute benefit seen in the soy vs. placebo meta-analysis. So it paints HRT as risky, and then promotes soy’s benefits with its definitive headline “Soy supplements can cool hot flashes” and other language in the story – without ever clearly defining the scope of the benefit.

What was the size of the benefit?  What was the size of the tradeoff of potential harms?  The story didn’t deliver that information – so a meaningful discussion of alternatives was not provided.  One could imagine a very helpful infographic being created to explain this.

To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better.”