Our Review Criteria

Please note: We stopped doing systematic criteria-driven reviews of news stories and PR news releases in December, 2018 when we lost all external funding.  But all past reviews remain accessible on the site. 

Between early 2006 and the end of 2018, we systematically reviewed 2,616 news stories. Between the start of 2015 until the end of 2018, we also systematically reviewed 606 public relations news releases as well. Click here for how those criteria slightly differ.

Health News Review CriteriaOur review criteria consist of 10 different elements that we think all health care news stories and all health care news (press) releases about interventions should include.

We think these criteria address the basic issues that consumers need to know in order to develop informed opinions about these interventions–and how/whether they matter in their lives. In some cases, it may be impossible or unreasonable for a story/release to address some of our criteria, in which case we’ll say so in our comments. We may rate certain such criteria as not applicable in some cases.

News Story Review Criteria

Criterion #1 Does the story adequately discuss the costs of the intervention?

News stories on costs of health care criteriaOur longterm experience with thousands of stories shows that the cost criterion gets the most unsatisfactory scores. News stories don’t help patients and consumers if they don’t tackle costs.
Read more.

Criterion #2 Does the story adequately quantify the benefits of the intervention?

New healthcare treatments should work well. When reading a story, people want to know: How effective is the intervention? Do the numbers back it up?
Read more. 

Criterion #3 Does the story adequately explain/quantify the harms of the intervention?

In the news, if you’re only hearing about the potential benefits of a test or treatment, run for the hills. What are the harms, and how often do they occur?
Read more.

Criterion #4 Does the story seem to grasp the quality of the evidence?

Ideally, healthcare interventions are subjected to rigorous testing to prove they work. When reading about a new intervention, device or diagnostic tool, people should come away with a sense of how rigorous the evidence is for the intervention.
Read more.

Criterion #5 Does the story commit disease-mongering?

Sometimes new conditions or diseases actually may just be opportunities to medicalize (and monetize) normal states of health. Readers aren’t well-served by stories that exaggerate or oversell conditions.
Read more.

Criterion #6 Does the story use independent sources and identify conflicts of interest?

Health care is rife with conflicts of interest. Readers deserve to know if the sources in the story have any conflicts of interest. They also deserve to hear the opinion of objective, independent third-party experts, who are vital to backstopping exaggerated claims.
Read more.

Criterion #7 Does the story compare the new approach with existing alternatives?

Just because something is “new” in health care doesn’t mean it’s better. When reading about a new drug, test, surgery, etc., readers should come away informed that there are other options on the table.
Read more.

Criterion #8 Does the story establish the availability of the treatment/test/product/procedure?

New medical treatments often get lots of attention before they’re ever made available to the public. After reading a story about a new intervention, a person should be able to answer: Is it actually available to me?
Read more.

Criterion #9 Does the story establish the true novelty of the approach?

Often something that’s lauded as new or novel really isn’t. Or it may be new, but it’s not better. So, when reading about a “new” medical intervention, people deserve to know: What is truly new and different about this idea?
Read more.

Criterion #10 Does the story appear to rely solely or largely on a news release?

Sometimes news stories rely solely or largely on a PR news release–and never disclose that the information came from a news release. Readers deserve to know if a story includes no original reporting.
Read more.


News Release Review Criteria

The 10 criteria used to review news releases are almost identical to those above.

Criterion #6 is slightly different for news release reviews. It gives this guidance:

Does the news release identify funding sources & disclose conflicts of interest?

Potential conflicts of interests should be disclosed in an editor’s note included in the release.  The funding for the project must also be disclosed along with any relationship that the  investigators may have with that funder.

Another significant difference for #6 regards the use of independent sources: Unlike news stories, news releases are not expected to feature comment from an objective expert.

The other key difference is Criterion #10.  That final criterion for news release reviews is this:

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Red flags rise up in our eyes when we see terms such as:

  • Public Relation word cloudbreakthrough
  • could become the new standard of care
  • cure
  • first-of-its-kind
  • game-changer
  • Holy Grail
  • magic
  • miracle
  • simple blood test
  • this might/may/could lead to (which means, of course, that “might/may/could not” applies as well)

Sources: Our criteria are at least partially influenced by, and appear in, the following seminal publications: