Joy Victory is deputy managing editor of HealthNewsReview.org. She tweets as @thejoyvictory.
This year, we reviewed nearly 200 health news stories using our 10-criteria system.
Reminder: We don’t review every kind of health news story. We only review stories that include claims about interventions: treatments, tests, products or procedures. And we only review stories reported by the set list of news organizations that we review regularly.
I joined the team in March, and hit the ground running serving as the third reviewer on news stories. In a bit, I’ll share my takeaways from reviewing hundreds of news stories.
But first, let’s look at the data.
We use a 5-star scoring system. Most of the news stories scored in the middle, with 87% of them earning 2, 3 or 4 stars.
Stories that earned a 5-star score were from the following sources, along with how many times they earned it.
While this can be a signal that these are high-quality sources, keep in mind that no source has ever earned 5-stars on every review we’ve given them. We see that quality is variable, even for the biggest and most well-regarded news sites.
And on the other end, here are the sources that earned 1-star reviews:
Again, this doesn’t mean the source is not to be trusted. Quality varies. And this isn’t a comprehensive look–we don’t keep reviewing sites that consistently earn one star. When it’s clear they aren’t going to make efforts to improve, we’d rather expend energy on sources motivated to improve.
Granted, we do not employ a random sample, and this may not be a perfect sample but we think it’s a representative sample. Indeed, it’s a bigger sample than anyone else has taken–we’ve reviewed 2,300 stories in 10 years, including about 200 this year. More on what we review, and how.
And in terms of traffic, our most popular news review was a look at a HealthDay story on the “Best Diets” rankings from U.S. News & World Report.
Now, as for what I’ve gleaned from looking at many, many news stories very, very closely? Let’s take a look:
Independent sources almost always make a story stronger. I’ve noticed that if a story does well on this criterion, it tends to do well on other criteria, too. But, if only a single source is interviewed–often the lead researcher on a study–then the story usually suffers in other ways. Important study limitations don’t get mentioned, viable alternatives to the treatment are overlooked, and availability of the intervention isn’t addressed, for example.
Headlines are often misleading. After joining the team, one of the first things I blogged about was how to write better health headlines. One common problem: Headlines that make a study’s outcome sound far more conclusive than it really was, as we examined in this post on “exercise lowers cancer risk” headlines.
As a former copy editor, I empathize with how hard it is to write a pithy and engaging headline around medical research. But, with a little extra thought, a headline can be both clicky and accurate, as we see from a few 5-star examples:
A well-written news release can beget a well-written news story. Last year we began reviewing news releases in addition to news stories, to see where “the pollution in the stream” begins, so to speak. It became apparent right away that a low-quality news release does impact news coverage–in fact, many times misleading statements are echoed in news stories.
But recently, we saw the opposite happen in this duo of studies on hallucinogenic mushrooms. In this case, the news releases and the news stories were all relatively well done.
We hope to see more of that in 2017!