Clear signs of improvement in health news grades in HealthNewsReview.org’s history

This is really interesting.  With our new site, we’re able to analyze data on the fly better than we’ve been able to do in the past.

We’ve now reviewed more than 1,600 stories.  A comparison of the first 800+ stories reviewed by HealthNewsReview.org with the most recent 800+ stories reviewed shows improvement in 8 of 10 categories.

The following is a table of the first 814 stories reviewed through June 2009.  The columns represent: the number of satisfactory scores, the number of unsatisfactory scores, the number of Not Applicable scores, and the percent of satisfactory scores when you subtract N/A scores from the total.  You can click on the criteria labels for a link to a reminder of what we’re looking for with those criteria.

Grades of 814 reviews:  from 2006 – June 2009

Criteria	# satisfactory	# unsatisfactory	# N/A	% satisfactory
Costs	200	512	102	28%
Benefits	228	573	13	28%
Harms	273	534	7	34%
Evidence	266	545	3	33%
Disease-mongering	550	241	23	70%
Sources/COI	427	385	2	53%
Alternatives	313	482	19	39%
Availability	552	225	37	71%
Novelty	657	129	28	84%
Rely on PR	516	20	278	96%

Now here is a comparable chart for the next 855 stories reviewed from July 2009 to the present.

Grades of 855 reviews: from July 2009 – December 2011

Criteria	# satisfactory	# unsatisfactory	# N/A	% satisfactory
Costs	214	488	153	30%
Benefits	321	509	25	39%
Harms	284	512	59	36%
Evidence	355	488	12	42%
Disease-mongering	662	143	50	82%
Sources/COI	471	371	13	56%
Alternatives	373	443	39	46%
Availability	562	183	108	75%
Novelty	595	196	64	75%
Rely on PR	644	84	127	88%

As you can see, grades improved on each of the first 8 criteria.

Interestingly, the two that didn’t improve both suggest the impact of news-release-driven news coverage.

• The number of stories that appeared to rely solely or largely on a news release went from 20 in the earlier time period (20/814 or 2.5%) to 84 in the later time period (84/855 or 9.8%).
• The percentage of stories that established the true novelty of the idea being reported decreased from 84% in the earlier time period to 75% in the later time period.  Many news-release-driven stories tend to promote an idea as novel even though it may not be all that novel an approach.

And don’t be thrown off by the seemingly high percentage of satisfactory scores on the news release criterion.  We must have clear evidence of text being copied directly from a news release into a story in order to rule a story unsatisfactory. Finding evidence that 104 out of 1,600 stories DID rely solely or largely on a news release is troubling.

But let’s focus on the positive:

• the 11 percentage point improvement over time in the rate of stories adequately quantifying benefits;
• the 9 percentage point improvement over time in the rate of stories evaluating the quality of the evidence;
• the 12 percentage point improvement over time in the rate of stories avoiding disease-mongering;
• the 7 percentage point improvement over time in the rate of stories comparing the new approach with existing alternatives.
• and average grades for four other criteria that all improved to a lesser degree.

We can never be sure if we were responsible for any of the improvement.  But we’re pleased to report the change, whatever the reasons.  And for this to happen during difficult economic times in many news organizations is even more impressive.

Meantime, let’s learn from the overall picture that is still too bleak.  In almost 6 years:

•  only 29% of stories satisfactorily discussed costs;
• only 34% of stories adequately quantified potential benefits – often a matter of failing to explain how small is the potential benefit;
• only 35% of stories adequately quantified potential harms – often a matter of failing to explain how large are the potential harms;
• only 37% of stories adequately evaluated the quality of the evidence they were reporting on;
• only 43% of stories compared the new approach with existing alternatives.

Glass half empty?  Or glass half full?  Either way, it’s a glass that’s becoming easier to see through clearly for what’s there and what’s not.