Posted by Gary Schwitzer in Health care journalism, Journal practices
Interesting post, pointing to an interesting paper published last week in the Annals of Emergency Medicine. Excerpt of the post:
Over 14 years, 84 editors at the journal rated close to 15,000 reviews by about 1,500 reviewers. Highlights of their findings:
…92% of peer reviewers deteriorated during 14 years of study in the quality and usefulness of their reviews (as judged by editors at the time of decision), at rates unrelated to the length of their service (but moderately correlated with their mean quality score, with better-than average reviewers decreasing at about half the rate of those below average). Only 8% improved, and those by very small amount.
How bad did they get? The reviewers were rated on a scale of 1 to 5 in which a change of 0.5 (10%) had been earlier shown to be “clinically” important to an editor.
The average reviewer in our study would have taken 12.5 years to reach this threshold; only 3% of reviewers whose quality decreased would have reached it in less than 5 years, and even the worst would take 3.2 years. Another 35% of all reviewers would reach the threshold in 5 to 10 years, 28% in 10 to 15 years, 12% in 15 to 20 years, and 22% in 20 years or more.So the decline was slow. Still, the results, note the authors, were surprising:
Such a negative overall trend is contrary to most editors’ and reviewers’ intuitive expectations and beliefs about reviewer skills and the benefits of experience.
Journalists should realize the flaws in the peer review system and perhaps reconsider a steady news diet that comes only or mostly from journal articles. And/or be prepared to discuss limitations more often.
Disclaimer: I welcome comments but will delete those with any kind of product pitch, profanity, personal attacks or those from anyone who doesn’t list what appears to be an actual e-mail address. I will also end any thread of comments that are repetitive. Because I moderate comments, I can’t keep reacting to repeatedly inaccurate or unsubstantiated claims. We don't give medical advice so we won't respond to questions asking for it.
Overall, this was an easy-to-follow summary. But there was a clear difference between this and a competing Boston Globe story, which did a better job offering context and analysis.
This Boston Globe blog post gave its audience a more thorough understanding of an emerging approach to virtual colonoscopy than the Wall Street Journal.
Longer, but overall not any better, than the coverage from the competing LA Times story. Not thorough enough to explain what this preliminary research actually means for readers–which is not much at this point.
We will not display your name or e-mail address on the site, nor will we share youre-mail address with any third parties.
For site updates and news from Gary, find us on Twitter:
HealthNewsReview: @HealthNewsRevu
Gary Schwitzer: @garyschwitzer
Copyright © 2012 Health News Review
Dan Cherkin posted on November 29, 2010 at 9:29 am
I have also noted that the quality of reviews, especially of NIH grant proposals, have deteriorated over the past decade. I wonder if this reflects greater difficulty in obtaining competent reviewers who are no longer willing to review because they have become overburdened responding to incompetent reviews (especially of grant proposals). Also, I believe that many reviewers, both competent and incompetent, are in a chronic state of over-commitment and are reducing the time they spend on lower priorities that have miniminal personal consequences for doing a poor job (e.g., reviewing manuscripts or grant proposals). Many such reviewers would be delighted if editors stopped sending them manuscripts to review!