Health News Review

This is really interesting.  With our new site, we’re able to analyze data on the fly better than we’ve been able to do in the past.

We’ve now reviewed more than 1,600 stories.  A comparison of the first 800+ stories reviewed by HealthNewsReview.org with the most recent 800+ stories reviewed shows improvement in 8 of 10 categories.

The following is a table of the first 814 stories reviewed through June 2009.  The columns represent: the number of satisfactory scores, the number of unsatisfactory scores, the number of Not Applicable scores, and the percent of satisfactory scores when you subtract N/A scores from the total.  You can click on the criteria labels for a link to a reminder of what we’re looking for with those criteria.

Grades of 814 reviews:  from 2006 – June 2009

Criteria  # satisfactory # unsatisfactory   # N/A  % satisfactory
Costs

200

512

102

28%

Benefits

228

573

13

28%

Harms

273

534

7

34%

Evidence

266

545

3

33%

Disease-mongering

550

241

23

70%

Sources/COI

427

385

2

53%

Alternatives

313

482

19

39%

Availability

552

225

37

71%

Novelty

657

129

28

84%

Rely on PR

516

20

278

96%

Now here is a comparable chart for the next 855 stories reviewed from July 2009 to the present.

Grades of 855 reviews: from July 2009 – December 2011

Criteria  # satisfactory # unsatisfactory   # N/A  % satisfactory
Costs

214

488

153

30%

Benefits

321

509

25

39%

Harms

284

512

59

36%

Evidence

355

488

12

42%

Disease-mongering

662

143

50

82%

Sources/COI

471

371

13

56%

Alternatives

373

443

39

46%

Availability

562

183

108

75%

Novelty

595

196

64

75%

Rely on PR

644

84

127

88%

As you can see, grades improved on each of the first 8 criteria.

Interestingly, the two that didn’t improve both suggest the impact of news-release-driven news coverage.

  • The number of stories that appeared to rely solely or largely on a news release went from 20 in the earlier time period (20/814 or 2.5%) to 84 in the later time period (84/855 or 9.8%).
  • The percentage of stories that established the true novelty of the idea being reported decreased from 84% in the earlier time period to 75% in the later time period.  Many news-release-driven stories tend to promote an idea as novel even though it may not be all that novel an approach.

And don’t be thrown off by the seemingly high percentage of satisfactory scores on the news release criterion.  We must have clear evidence of text being copied directly from a news release into a story in order to rule a story unsatisfactory. Finding evidence that 104 out of 1,600 stories DID rely solely or largely on a news release is troubling.

But let’s focus on the positive:

  • the 11 percentage point improvement over time in the rate of stories adequately quantifying benefits;
  • the 9 percentage point improvement over time in the rate of stories evaluating the quality of the evidence;
  • the 12 percentage point improvement over time in the rate of stories avoiding disease-mongering;
  • the 7 percentage point improvement over time in the rate of stories comparing the new approach with existing alternatives.
  • and average grades for four other criteria that all improved to a lesser degree.

We can never be sure if we were responsible for any of the improvement.  But we’re pleased to report the change, whatever the reasons.  And for this to happen during difficult economic times in many news organizations is even more impressive.

Meantime, let’s learn from the overall picture that is still too bleak.  In almost 6 years:

  •  only 29% of stories satisfactorily discussed costs;
  • only 34% of stories adequately quantified potential benefits – often a matter of failing to explain how small is the potential benefit;
  • only 35% of stories adequately quantified potential harms – often a matter of failing to explain how large are the potential harms;
  • only 37% of stories adequately evaluated the quality of the evidence they were reporting on;
  • only 43% of stories compared the new approach with existing alternatives.

Glass half empty?  Or glass half full?  Either way, it’s a glass that’s becoming easier to see through clearly for what’s there and what’s not.

Comments

e-Patient Dave posted on December 13, 2011 at 12:55 pm

I love this! Prospective tracking, if you will, of story quality.

Two suggestions for next steps:

- Can you find a statistical-significance geek to say which of the improvements is better than p=.05, or some such? For instance, the harms shift is only a dozen out of hundreds. (Might as well model the self-scrutiny that we encourage others to do!)

- Any indication of whether individual editors or writers are aware of HNR? It would be awesome if you could trace a “smoking gun” connection between the service and the improvement.

Hm, also – you tally quality by publisher – is it easy to identify individual superstar writers? Might be awesome to give them an award at an NSA convention – great publicity for a very legitimate reason!