NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine -
Read Original Story

Soy Supplements Can Cool Hot Flashes: Study


2 Star

Soy Supplements Can Cool Hot Flashes: Study

Our Review Summary

We don’t think this story provided answers to women, as it failed on most of our criteria – all of which are intended to address elements of a story that we think consumers need in order to evaluate studies and claims about health care interventions.


Why This Matters

We need to help readers evaluate the quality of evidence.

What does it mean when a meta-analysis looks at studies of soy versus placebo?

What might be the limitations of such an approach?

What have we done to make news consumers/health care consumers any smarter by this story?

Not much, we’re afraid.


Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The story dances all around – but never delivers specific data on the absolute benefit seen in the meta-analysis.

The closest it comes is this:  “soy supplements with higher amounts of the isoflavone called genistein were more than twice as good at reducing hot flash frequency than those with low amounts.”

And this non-evidence statement from one of the author-researchers: “What this study shows is that ingesting soy isoflavones will help you. I personally think foods [containing soy] are better.”

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

No discussion of potential harm.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

The study on which the story is based is a meta-analysis, which combines multiple studies that, taken alone, may fail to show any difference between treatments, to see if the pooled results provide greater statistical power to detect a small difference that might be missed in a single study.  If done properly, a meta-analysis can provide useful insights.  But the statistical methods are complex and the devil is in the details.

At the end, an independent expert briefly raises this point – “Because the studies vary greatly in their methods and the preparations used, we still don’t know who are women who will benefit.”

But by then the story had driven home the suggestion of definitive benefit – in the headline and in ensuing copy – and it was too little too late.

To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better” – but with no data provided to support that opinion.

Does the story commit disease-mongering?

Not Applicable

No disease-mongering, but, then again, there wasn’t much information about hot flashes at all in the story.

Does the story use independent sources and identify conflicts of interest?


Perhaps the only saving grace of the story:  One clearly skeptical independent expert and one other who noted limitations.

Does the story compare the new approach with existing alternatives?

Not Satisfactory

The story only stated, “Soy has been touted as an alternative treatment to hormone replacement therapy after HRT was linked to an increased risk of breast cancer.”

But there’s still a decision to be made by women – taking both harms and benefits into account.  As already noted in the “Benefits” criterion comment above, the story never delivers specific data on the absolute benefit seen in the soy vs. placebo meta-analysis. So it paints HRT as risky, and then promotes soy’s benefits with its definitive headline “Soy supplements can cool hot flashes” and other language in the story – without ever clearly defining the scope of the benefit.

What was the size of the benefit?  What was the size of the tradeoff of potential harms?  The story didn’t deliver that information – so a meaningful discussion of alternatives was not provided.  One could imagine a very helpful infographic being created to explain this.

To make matters worse, both the subhead and the body copy allow a researcher to go beyond the evidence of the study – which was about soy supplements – and to offer her opinion, “I personally think foods containing soy are better.”

Does the story establish the availability of the treatment/test/product/procedure?

Not Applicable

The availability of soy supplements is not in question.

Does the story establish the true novelty of the approach?


The story at least quoted another researcher, whose own work “published in 2011 in the Archives of Internal Medicine, she found no effects on menopausal symptoms when women took 200 mg of soy isoflavones for two years.”

Does the story appear to rely solely or largely on a news release?


Because two independent experts were quoted, we don’t think there’s any evidence that the story relied on a news release.

Total Score: 3 of 8 Satisfactory

Comments (1)

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.

Liz Scherer

April 16, 2012 at 10:57 am

Here’s my take on the meta-analysis, with links to other data.

The trouble with soy reports is that the compounds used in studies are dissimilar and it’s almost impossible to accurately highlight costs. Still, I am happy that you covered this.