This is really interesting. With our new site, we’re able to analyze data on the fly better than we’ve been able to do in the past.
We’ve now reviewed more than 1,600 stories. A comparison of the first 800+ stories reviewed by HealthNewsReview.org with the most recent 800+ stories reviewed shows improvement in 8 of 10 categories.
The following is a table of the first 814 stories reviewed through June 2009. The columns represent: the number of satisfactory scores, the number of unsatisfactory scores, the number of Not Applicable scores, and the percent of satisfactory scores when you subtract N/A scores from the total. You can click on the criteria labels for a link to a reminder of what we’re looking for with those criteria.
Grades of 814 reviews: from 2006 – June 2009
Criteria | # satisfactory | # unsatisfactory | # N/A | % satisfactory |
Costs |
200 |
512 |
102 |
28% |
Benefits |
228 |
573 |
13 |
28% |
Harms |
273 |
534 |
7 |
34% |
Evidence |
266 |
545 |
3 |
33% |
Disease-mongering |
550 |
241 |
23 |
70% |
Sources/COI |
427 |
385 |
2 |
53% |
Alternatives |
313 |
482 |
19 |
39% |
Availability |
552 |
225 |
37 |
71% |
Novelty |
657 |
129 |
28 |
84% |
Rely on PR |
516 |
20 |
278 |
96% |
Now here is a comparable chart for the next 855 stories reviewed from July 2009 to the present.
Grades of 855 reviews: from July 2009 – December 2011
Criteria | # satisfactory | # unsatisfactory | # N/A | % satisfactory | |
|
214 |
488 |
153 |
30% |
|
Benefits |
321 |
509 |
25 |
39% |
|
Harms |
284 |
512 |
59 |
36% |
|
Evidence |
355 |
488 |
12 |
42% |
|
Disease-mongering |
662 |
143 |
50 |
82% |
|
Sources/COI |
471 |
371 |
13 |
56% |
|
Alternatives |
373 |
443 |
39 |
46% |
|
Availability |
562 |
183 |
108 |
75% |
|
Novelty |
595 |
196 |
64 |
75% |
|
Rely on PR |
644 |
84 |
127 |
88% |
As you can see, grades improved on each of the first 8 criteria.
Interestingly, the two that didn’t improve both suggest the impact of news-release-driven news coverage.
And don’t be thrown off by the seemingly high percentage of satisfactory scores on the news release criterion. We must have clear evidence of text being copied directly from a news release into a story in order to rule a story unsatisfactory. Finding evidence that 104 out of 1,600 stories DID rely solely or largely on a news release is troubling.
But let’s focus on the positive:
We can never be sure if we were responsible for any of the improvement. But we’re pleased to report the change, whatever the reasons. And for this to happen during difficult economic times in many news organizations is even more impressive.
Meantime, let’s learn from the overall picture that is still too bleak. In almost 6 years:
Glass half empty? Or glass half full? Either way, it’s a glass that’s becoming easier to see through clearly for what’s there and what’s not.
Comments
Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.
e-Patient Dave
December 13, 2011 at 12:55 pmI love this! Prospective tracking, if you will, of story quality.
Two suggestions for next steps:
– Can you find a statistical-significance geek to say which of the improvements is better than p=.05, or some such? For instance, the harms shift is only a dozen out of hundreds. (Might as well model the self-scrutiny that we encourage others to do!)
– Any indication of whether individual editors or writers are aware of HNR? It would be awesome if you could trace a “smoking gun” connection between the service and the improvement.
Hm, also – you tally quality by publisher – is it easy to identify individual superstar writers? Might be awesome to give them an award at an NSA convention – great publicity for a very legitimate reason!
Our Comments Policy
But before leaving a comment, please review these notes about our policy.
You are responsible for any comments you leave on this site.
This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.
We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.
We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.
There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”
The authors of the Retraction Watch comments policy urge commenters:
We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.
And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.
The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.
You might also like