Clear signs of improvement in health news grades in HealthNewsReview.org’s history

This is really interesting.  With our new site, we’re able to analyze data on the fly better than we’ve been able to do in the past.

We’ve now reviewed more than 1,600 stories.  A comparison of the first 800+ stories reviewed by HealthNewsReview.org with the most recent 800+ stories reviewed shows improvement in 8 of 10 categories.

The following is a table of the first 814 stories reviewed through June 2009.  The columns represent: the number of satisfactory scores, the number of unsatisfactory scores, the number of Not Applicable scores, and the percent of satisfactory scores when you subtract N/A scores from the total.  You can click on the criteria labels for a link to a reminder of what we’re looking for with those criteria.

Grades of 814 reviews:  from 2006 – June 2009

Now here is a comparable chart for the next 855 stories reviewed from July 2009 to the present.

Grades of 855 reviews: from July 2009 – December 2011

As you can see, grades improved on each of the first 8 criteria.

Interestingly, the two that didn’t improve both suggest the impact of news-release-driven news coverage.

• The number of stories that appeared to rely solely or largely on a news release went from 20 in the earlier time period (20/814 or 2.5%) to 84 in the later time period (84/855 or 9.8%).
• The percentage of stories that established the true novelty of the idea being reported decreased from 84% in the earlier time period to 75% in the later time period.  Many news-release-driven stories tend to promote an idea as novel even though it may not be all that novel an approach.

And don’t be thrown off by the seemingly high percentage of satisfactory scores on the news release criterion.  We must have clear evidence of text being copied directly from a news release into a story in order to rule a story unsatisfactory. Finding evidence that 104 out of 1,600 stories DID rely solely or largely on a news release is troubling.

But let’s focus on the positive:

• the 11 percentage point improvement over time in the rate of stories adequately quantifying benefits;
• the 9 percentage point improvement over time in the rate of stories evaluating the quality of the evidence;
• the 12 percentage point improvement over time in the rate of stories avoiding disease-mongering;
• the 7 percentage point improvement over time in the rate of stories comparing the new approach with existing alternatives.
• and average grades for four other criteria that all improved to a lesser degree.

We can never be sure if we were responsible for any of the improvement.  But we’re pleased to report the change, whatever the reasons.  And for this to happen during difficult economic times in many news organizations is even more impressive.

Meantime, let’s learn from the overall picture that is still too bleak.  In almost 6 years:

•  only 29% of stories satisfactorily discussed costs;
• only 34% of stories adequately quantified potential benefits – often a matter of failing to explain how small is the potential benefit;
• only 35% of stories adequately quantified potential harms – often a matter of failing to explain how large are the potential harms;
• only 37% of stories adequately evaluated the quality of the evidence they were reporting on;
• only 43% of stories compared the new approach with existing alternatives.

Glass half empty?  Or glass half full?  Either way, it’s a glass that’s becoming easier to see through clearly for what’s there and what’s not.