How We Rate Stories

HealthNewsReview.org only reviews news stories and news releases that include a claim of efficacy about:

  • Specific treatments, tests, products or procedures (A story that includes multiple claims about multiple interventions or for multiple uses may not be eligible for review because our review criteria can generally only be applied to the review of a single, specific intervention for a single, specific condition.) So the stories may be about:
    • Drugs or devices
    • Vitamins or nutritional supplements
    • Diagnostic and screening tests
    • Dietary recommendations
    • Surgical procedures
    • Psychotherapy/mental health interventions

What are the review criteria and how are they applied?

Our review criteria consist of 10 different elements that we think all health care news stories about interventions should include. (A slightly modified list of criteria are used in reviewing health care news releases.) We think these criteria address the basic issues that consumers need to know in order to develop informed opinions about these interventions – and how/whether they matter in their lives.  In some cases, it may be impossible or unreasonable for a story to address some of our criteria, in which case we’ll say so in our comments.  We may rate certain such criteria as not applicable in some cases.

Each of the criteria are at least partially influenced by, and appear in, the following seminal publications:

Click here to view review criteria


Who are the reviewers?

A multi-disciplinary team of reviewers from journalism, medicine, health services research, public health and patient perspectives assesses the quality of the stories and news releases using a standardized rating system. Stories and news releases are graded and critiques are published on this website. Two or three different reviewers evaluate each story.  Hundreds of years of experience are represented on this team of expert reviewers.

Read more about our reviewers.


Which news organizations are reviewed?

News stories will be monitored each Monday through Friday in major U.S. media. We may, at any time, review any news story we see from any news organization.

Over our entire history, starting in 2006, we have changed the scope of whom we review on several occasions.

As of January 5, 2015, the following are the media we will review regularly. We will only review what we find on the websites of these news organizations.  On many days we will not be able to review all of the eligible stories that we find.  In such instances, we will decide what to review based on a number of factors, including:

  • deferring to news organizations that we have reviewed less frequently or less recently
  • deferring to stories on topics that apply to a broader population base

Each day we will review the websites of the following papers that are among the leaders in daily circulation:

  • Boston Globe
  • Los Angeles Times
  • New York Times
  • Philadelphia Inquirer
  • USA Today
  • Wall Street Journal
  • Washington Post

Also checked daily:

  • National Public Radio health & science page
  • The websites of the following TV networks: ABC, CBS, CNN, Fox, NBC

Wire or news services:

  • Associated Press health news
  • Bloomberg News
  • HealthDay
  • Reuters Health

Checked regularly

  • Websites of news magazines TIME, Newsweek, U.S. News & World Report
  • Vox.com
  • Slate.com
  • FiveThirtyEight.com
  • BuzzFeed.com

NOTE:

Because we have changed whom we review several times over our history, because of the volume of eligible stories (not all of which we can review on any given day), and because of the limitations of our staffing and funding, we cannot claim that our sampling is representative of any individual’s work nor of any individual news organization. For example, in our first 9 years, we “only” reviewed140 stories by the New York Times.  Nonetheless, it is difficult to argue with a database of 2,000 news stories and what it shows about overall industry problems that could be easily corrected.

During our first 3.5 years, we reviewed ABC, CBS and NBC health news stories every day. In 2009, we decided to stop this very time-consuming process of having three reviewers apply ten standardized criteria to TV news stories because, frankly, overall the stories were not improving. We have limited resources and can’t review everything. We chose to apply our time, resources and energy to new and different media with whom we might have more impact. That does not mean we are ignoring TV health news.  In fact, since that decision in 2009 we have commented frequently on our blog about TV health news.


How is the star score determined?

For each news story or news release, the 10 criteria will be scored as “Satisfactory,” “Unsatisfactory” or “Not Applicable.” Total scores are posted for articles that have three or fewer “not applicable” ratings, and are expressed as proportions. For the purposes of display, the total scores are translated into a star rating.

Percent of criteria judged satisfactory Star score assigned
0% 0 stars
1-20% 1 star
21-40% 2 stars
41-60% 3 stars
61-80% 4 stars
81-100% 5 star 

(If one or two items are rated ‘NA’, the denominator will be 8 or 9 rather than 10.)


Does Health News Review give medical advice?

HealthNewsReview.org does not provide medical opinion. Nothing in our reviews should be construed as medical advice.


Is HealthNewsReview.org fair to journalists?

We understand there are many challenges to achieving accuracy, balance and completeness in health care journalism. For a reporter, there are deadlines, editors, and corporate financial pressures. We hold the bar high for quality in health care journalism because it plays a major role in educating consumers. Consumers need to be well informed to make sound choices that affect their health and well being.

We strive to give the journalist the benefit of the doubt in our review comments.  And we always try to offer constructive criticism, making suggestions whenever possible about how an approach could have been improved.



New in April 2015

We are excited to add systematic reviews of health care-related news releases, written for medical journals, drug/device and other industry representatives, hospitals, academic medical centers and others.

We’ve added new reviewers to take on this task, with a process that is very similar to our time-honored approach with reviewing health care news stories.