Our Review Criteria

Excellent-Grade-iStock_000048229824_350x285After reviewing more than 2,000 news stories about health care interventions over the past 10 years, we have found a clear pattern of imbalanced media messages – often exaggerating or emphasizing potential benefits, minimizing or ignoring potential harms, and ignoring cost issues.  That’s a terrible triad if one hopes for an informed health care consumer population. Our review criteria consist of 10 different elements that we think all health care news stories and all health care news (press) releases about interventions should include.  We think these criteria address the basic issues that consumers need to know in order to develop informed opinions about these interventions – and how/whether they matter in their lives. The average grade on 5 of the 10 criteria below is a failing grade (>55% unsatisfactory). In some cases, it may be impossible or unreasonable for a story/release to address some of our criteria, in which case we’ll say so in our comments.  We may rate certain such criteria as not applicable in some cases.

Each of the criteria are at least partially influenced by, and appear in, the following seminal publications:

News Story Review Criteria

(news release review criteria appear lower on the page)

Criterion #1 Does the story adequately discuss the costs of the intervention?

About 70% of the stories we review fail to satisfy this criterion.


  • It”s not good enough to say, “The cost is much lower than the invasive procedures.” What is that cost? How much lower?
  • Do insurance policies cover the intervention?
  • Might there be costs for copays and other expenses (psychologists, dietitians, etc.) that amount to significant additional expenditure? If you can”t quantify them you can at least mention them.
  • It may be difficult to estimate costs of an experimental approach early in its development. But a media message can at least cite costs of existing alternatives.  If the new approach is comparable to other approaches then the cost of the alternative methods could be cited. Our rule of thumb: If it’s not too early to talk about how well something might work, then it”s not too early to start discussing what it may cost.

Read More…

Criterion #2 Does the story adequately quantify the benefits of the treatment/test/product/procedure?

We expect stories to give readers some sense – a quantitative estimate – of the scope of the potential benefit. Many stories:

  • Use only relative, not absolute, risk reduction or benefit data.
  • Rely too heavily on what may be unrepresentative patient anecdotes about benefits.
  • Cite statistics appropriately, but allow even a single unchallenged exaggerated quote to throw the story out of balance


Read More…

Criterion #3 Does the story adequately explain/quantify the harms of the intervention?

Screen-Shot-2015-04-10-at-8.01.08-PMMany stories emphasize or exaggerate potential benefits while minimizing or completely ignoring potential harms. They may:

  • Fail to mention potential harms.
  • Fail to quantify potential harms.
  • Fail to describe the severity of potential harms.
  • Fail to account for “minor” side effects that could have a significant impact on a patient’s life.
  • Rely too heavily on a patient anecdote about safety.
  • Rely too heavily on a researcher’s comment that an approach appears to be safe – without supporting data.

Read More…

Criterion #4 Does the story seem to grasp the quality of the evidence?

We expect journalists to critically evaluate the evidence, not merely to take published or presented research as gospel. Many stories:

  • Fail to point out the limitations of the evidence.
  • Fail to include a caution about interpretation of uncontrolled data.
  • Fail to point out if a primary outcome is a surrogate marker or fail to caution readers/viewers about extrapolating this to health outcomes.
  • Fail to point out the limited peer review that may have taken place with findings presented at a scientific meeting.
  • Present anecdotes as evidence of a treatment’s harms or benefits – rather than as a single illustration of its use.


Read More…

Criterion #5 Does the story commit disease-mongering?

Screen-Shot-2015-04-10-at-8.06.21-PMThis criterion assesses whether a story exaggerates or over-sells a condition. There are different forms of “mongering” – turning risk factors into diseases (e.g., low bone mineral density becomes osteoporosis); misrepresentation of the natural history and/or severity of a disease (e.g., early-stage low-grade prostate cancer); medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction or female ‘sexual dysfunction’); medicalization of normal states (baldness, wrinkles, shyness, menopause); or exaggeration of the prevalence of a disorder (e.g., using rating scales to ‘diagnose’ erectile dysfunction).

Read More…

Criterion #6 Does the story use independent sources and identify conflicts of interest?

In order to get a satisfactory score on this criterion, two things must occur: There must be an independent expert source quoted (someone not directly connected with the research) and there must be some attempt to let the public know about potential conflicts of interest.


Read More…

Criterion #7 Does the story compare the new approach with existing alternatives?

Screen-Shot-2015-04-10-at-8.08.10-PMWe understand that news wants to cover what”s new. But we think there”s an obligation to put the new into the context of existing alternatives that have a longer, more proven track record. Unsatisfactory stories may:

  • Discuss a surgical approach without mentioning non-surgical alternative approaches.
  • Discuss a new test without mentioning other tests that are available, including the option of not being screened in the case of a screening test.
  • Fail to discuss the advantages/disadvantages of the new idea compared with existing approaches.
  • Fail to discuss how the new treatment, test, product or procedure fits into the realm of existing alternatives.

Read More…

Criterion #8 Does the story establish the availability of the treatment/test/product/procedure?

Screen-Shot-2015-04-10-at-8.08.59-PMDoes the story make a drug sound like it”s available at the corner drug store (or may be soon) when it”s really only in an early-phase trial? Does it feature one doctor promoting a technology without providing a sense of how many trained doctors there are, or what the learning curve is? Did it allow promoters to make unfounded predictions of FDA approval or widespread availability? There needs to be some sense of history of the thousands of previous flashes in the pan in medical research that were equally exciting but never panned out.

Read More…

Criterion #9 Does the story establish the true novelty of the approach?

Screen-Shot-2015-04-10-at-8.11.12-PMMany “new” treatments, tests, products or procedures are not really novel. The agent may be another member of a well-established therapeutic class of drug; even if it represents a new class, it may offer no more than the drugs that are widely available. In the press release for a new drug this essential information may be lost in the hype and the drug is portrayed as “novel” in order to increase initial sales. We believe that journalists should be able to make these distinctions by doing their research or by asking independent sources.

Read More…

Criterion #10 Does the story appear to rely solely or largely on a news release?

It is simply not sound journalistic practice to lift material directly from a news release without stating that this is the case. This criterion will rarely be employed. If, on the front end, we know that a news release was sent out and if we can get a copy of that release, we will review it along with the story.


  • For a story to be rated unsatisfactory, we must have clear evidence that the exact same wording from a news release appears in a story.
  • Lacking such evidence, the worst a story can rate is N/A.
  • With a story that has several independent sources, we can safely assume that it did NOT rely on a news release and thus, give it a Satisfactory score.

Read More…


News Release Review Criteria

The 10 criteria used to review news releases are almost identical to those above.

Criterion #6 is slightly different for news release reviews. It gives this guidance:

Does the news release identify funding sources & disclose conflicts of interest?

Potential conflicts of interests should be disclosed in an editor’s note included in the release.  The funding for the project must also be disclosed along with any “official” tie investigators may have with that funder.

Another significant difference for #6 regards the use of independent sources: unlike news stories, news releases are not expected to feature comment from an objective expert.

The other key difference is Criterion #10.  That final criterion for news release reviews is this:

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Red flags rise up in our eyes when we see terms such as:

  • breakthrough
  • could become the new standard of care
  • cure
  • first-of-its-kind
  • game-changer
  • Holy Grail
  • magic
  • miracle
  • simple blood test
  • this might/may/could lead to…. (which means, of course, that “might/may/could not” applies as well)