News releases exaggerate the results of scientific research — a new study shows why this should stop

The following post is by Matt Shipman, a public information officer at North Carolina State University and board member of the National Association of Science Writers. He’s also a frequent story and news release reviewer for HealthNewsReview.org. He tweets as @shiplives.

exaggeration

Even news releases from respected peer-reviewed journals are subject to exaggeration – but those exaggerations aren’t associated with significant increases in news coverage of the research being presented.

Those are the findings of a recent paper published in the journal PLOS ONE that highlights the important role that press officers play in shaping health and science news – and the need for reporters to avoid relying solely on news releases when writing about research findings.

The paper presents findings of a study that evaluated the extent to which news releases from science and medical journals exaggerate research findings – or provide caveats related to those findings. In addition, the study assessed whether the presence of exaggerated findings, or caveats about those findings, influenced news coverage of the research.

For the study, researchers from the United Kingdom and Australia looked at 534 news releases issued by prominent journals, including the Lancet, BMJ, Nature and Science. The research team also analyzed the 534 journal articles associated with those releases, and 582 news stories that covered the research findings.

Many of the authors of the PLOS ONE article also were part of the research team that published a 2014 paper in BMJ that addressed similar questions regarding research-related news releases from 20 universities in the U.K. HealthNewsReview.org’s publisher, Gary Schwitzer, wrote about that paper when it came out.

The paper addresses exaggerations and caveats separately, so let’s start with the exaggerations.

Exaggerated findings don’t receive more news coverage

The researchers reported that 23 percent of news releases contained more direct or explicit advice than was found in their relevant journal articles (i.e. exaggerations), and that 21 percent of releases exaggerated causal statements (i.e. they made cause-and-effect claims for research that can’t prove such a relationship).

And those exaggerations were echoed in news stories.

The odds that a news story would include exaggerated advice were 2.4 times higher for studies whose news releases included exaggerated advice. And the odds that a news story would use stronger causal language than the related journal article were 10.9 times higher if the news release also contained exaggerated causal claims.

But the findings don’t offer any strong evidence that exaggerations were associated with more pickup from news outlets. In terms of advice, 53 percent of news releases without exaggerated advice (101 out of 191) secured news coverage, whereas 34 percent of news releases with exaggerated advice (19 of 56) garnered news. In regard to causal claims, 49 percent of releases that didn’t include exaggerations (63 of 129) got news coverage, compared to 66 percent of releases that did include exaggerations (23 of 35).

However, these differences were not statistically significant. Taking into account the smaller number of releases that included exaggerations of either kind – limiting the sample size – the most cautious way to interpret these numbers is to say that exaggerations in news releases don’t seem to make much of a difference one way or the other on whether reporters will cover the research.

But is that conclusion fair?

The recent PLOS ONE study, and the 2014 BMJ study, both focused solely on U.K. news outlets, and the 2014 study looked only at releases from U.K. universities, so readers should be cautious about extrapolating the findings to institutions and news outlets in other parts of the world.

Moreover, as the study authors very appropriately point out: “It is always possible that some of the apparently causal links are circumstantial or mediated by factors we did not analyse. For example the rare use of caveats might potentially have co-occurred with unknown other characteristics influencing news uptake in order to disguise any caveat influence. The possibility of unknown correlated factors is generally true for correlational research.”

Why are news release exaggerations likely to find their way into news stories?

For one thing, reporters are often tasked with turning around stories quickly – giving them little time to do the reporting needed to identify exaggerations.

“We know that the news industry has changed over the years, which has put pressure on journalists to produce more stories in less time on topics that they may not have a background in,” says Jen LaLoup, editorial media manager for PLOS, which publishes PLOS ONE. “It’s unfortunate, but if a journalist relies on a press release as their primary source for their story, and doesn’t have the time to fully report on the research, it is likely that they will repurpose the release with the exaggerations and caveats included. Which means, now more than ever, it is important for press officers to minimize exaggerations in press releases.”

“When we think about the news supply chain, reporters start with the press release,” says Dr. Preeti Malani, an associate editor at JAMA – and former reporter – who has helped oversee the media relations team at the JAMA Network. “Most (the good ones) are going to spend time on the actual article too and hopefully confirm or temper what is written in a press release (which is designed to generate attention). That said, the overall decrease in the number of reporters (and the specific drop in health reporters) means these press releases are sometimes published verbatim.”

In other words, some exaggerations in news stories are likely not being included intentionally by the reporters. And, by the same token, many public information officers (PIOs; communications/public relations officials) suggest, exaggerations found in news releases may not stem from deliberate attempts to mislead readers or to generate attention based on hype.

How could exaggerations be making their way into news releases?

Let’s not be naïve: Sometimes the exaggeration found in news releases may well be put there intentionally.

“It has been my experience that a majority of PIOs want to make sure research findings are represented accurately, but we do see instances where there is pressure to write about research in a better light than is warranted,” LaLoup says.

“I am certain some PIOs feel pressured to oversell findings,” adds Angela Hopp, communications director at the American Society for Biochemistry and Molecular Biology (which publishes several journals) – and a former science writer at the University of Houston.  “If a manager or someone even higher up the food chain creates an environment of desperation, that will affect the staff’s decision-making.”

However, Terry Devitt, director of research communications at the University of Wisconsin-Madison, stresses, “Common sense and preserving credibility dictate sticking to results without undue embellishment – including caveats and disclaimers and noting any potential conflicts is important.”

The paper notes that many public information officers (PIOs) who write news releases go out of their way to avoid exaggerating findings. And for good reason.

Tips for avoiding exaggeration

avoiding exaggeration

Our toolkit includes a host of resources for journalists, press officers, and consumers who want to understand how to evaluate research studies and the claims they generate. Take a look at what’s available here.

“I think if you need to exaggerate a result to get attention you either are not a very good press officer or the research isn’t very good,” Devitt says. “It can also be very risky to go beyond what the data say. If, for example, you are writing about a condition like autism where you have very large and engaged audiences, exaggeration can inspire some serious blowback as those audiences tend to be well informed and have the motivation and tools to dig deep. You can get into some serious trouble trying to make more of a result than what a reasonable interpretation of the data shows. A careless or venal press officer can also be excoriated by watchdogs like HealthNewsReview.org. Who wants that kind of experience?”

Exaggeration may be a side effect of trying to write more simply and clearly

And the PLOS ONE paper presents another possibility for how exaggerations may find their way into news releases: “Note that the types of exaggeration studied here are quite subtle and routine: for example changing a phrase like ‘related to’ or ‘might increase’ to a direct causal phrase like ‘boosts.’ We have no evidence that these small exaggerations (or message creep) represent conscious efforts to hype….Unintended subtle exaggerations may arise for other reasons such as trying to use simpler and more direct language with fewer words per sentence.”

[Editor’s note: Contrary to the study authors’ suggestion, we would consider “might increase” to be a causal phrase that should be avoided in reports about observational research. More on that issue here.]

This situation could be exacerbated by staffing issues.

“[I]f a press office is understaffed or inexperienced, the tough questions won’t get asked and overselling won’t get flagged,” Hopp says.

“I think this is one of the most important points of the paper, even if it’s not something that their data directly address,” says Quinn Eastman, a science writer at Emory University’s Woodruff Health Sciences Center. Eastman notes that this sort of analysis can provide real value to PIOs: “To avoid exaggerations, PIOs can do like doctors do: make checklists of pitfalls, and practice steering around them. Rewrite by taking a different approach.”

[Editor’s note: we think HealthNewsReview.org’s review criteria are a pretty handy checklist for avoiding exaggeration. There are more suggestions in the sidebar (above right).]

Universities vs. journals: the environment may make a difference  

Press officers at universities may face specific challenges when crafting news release messages compared with their counterparts at science journals.

“University PIOs have so many audiences,” Hopp says. “[University PIOs are] writing news releases to be sent to journalists, and those releases also must resonate with folks who visit the university website – donors, alumni, students, faculty, community members, local and state leaders and so forth. They’re adapting those releases into stories for campus newsletters, magazines and multimedia. And they’re often covering lots of schools on campus all at once. You’re running across campus every day to interview people and trying to chase down experts with members of the media in tow.

“Journal PIOs, meanwhile, are writing releases primarily for journalists,” Hopp says. “But a major secondary audience for releases by member organizations is the membership itself. Journal PIOs have a predictable schedule in terms of when new issues come out. And, while they’re likely pitching in on tons of writing projects not directly related to the journal, they’re not physically on the go all the time.”

And that lack of predictability for university PIOs can exacerbate the need to write a news release about complex research in a very short time frame. Couple that with writing for a wide variety of audiences, and the potential for an inadvertent exaggeration increases. This may explain why the U.K. university news releases contained more exaggerations than the journal news releases.

However, these factors don’t excuse the exaggerations – they are still a problem, regardless of whether they were incorporated into a news release intentionally or not. But understanding how they may happen can help us find ways to avoid having them happen in the future. Which brings us to caveats…

Including caveats isn’t linked to decreased news coverage

To assess whether including caveats in a news release was associated with changes in news coverage, the researchers combined the data from the 2014 study of U.K. university news releases and the study of releases from research journals.

As with exaggerations, the researchers found “no evidence that caveats reduce news uptake.” But the researchers did find that incorporating caveats into news releases was associated with more cautious news coverage.

News stories were 9.5 times more likely to include caveats about advice if a news release included a caveat. Similarly, news stories were 7.6 times more likely to include caveats about causal claims if the news release contained a caveat.

“[T]he finding that caveats don’t decrease coverage is encouraging,” says LaLoup. “This study suggests journalists aren’t ignoring the caveats in releases, and that should help reassure press officers that they should include them in their press releases.”

Is this useful for press officers?

The first reaction from some PIOs is to be defensive, and to feel like this is another example of people trying to pin the blame for imperfect news coverage on institutional science writers.

I don’t think that’s true.

News outlets are ultimately responsible for the content of their stories. Journalism is not simply regurgitating facts (or news releases). It involves critical thinking, analysis and, well, reporting – talking to multiple sources and assessing all of the available information before writing a story.

More importantly, I think the PLOS ONE paper gives PIOs tools that can help them get better at their jobs. [Note: I’m a PIO, and I found the paper useful.]

And I’m not alone.

“It’s good to know that including elements like ‘this result was in an animal model/small clinical trial, but needs to be tested in a bigger way’ does not discourage news coverage,” Eastman says.

I’d add that the language regarding how exaggerations can inadvertently make their way into news releases is valuable: the more people are aware of how this can happen, the more alert they can be to spotting such exaggerations and removing them before a news release is issued (or counteracting them afterward in the event that such statements do make their way into the final public news release).

Takeaways for journalists, PIOs, and the public 

Ultimately, what I took from this paper is a renewed sense of the important role that PIOs play in accurately portraying science and health news to the public. When you’re talking about human health, the stakes are high. That means there is a burden of responsibility on PIOs, reporters and researchers to communicate clearly and honestly with the public. The more we understand about this process, the better we are able to meet that responsibility head on.

You might also like

Comments (1)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Stephen Cox, MD

December 28, 2016 at 4:21 pm

Exaggerated benefits and risks are often due to reporters and even peer review articles NOT distinguishing between relative and absolute changes. This allows a great amount of misinformation and confusion.