Our review criteria consist of 10 different elements that we think all health care news stories and all health care news (press) releases about interventions should include. We think these criteria address the basic issues that consumers need to know in order to develop informed opinions about these interventions – and how/whether they matter in their lives. In some cases, it may be impossible or unreasonable for a story/release to address some of our criteria, in which case we’ll say so in our comments. We may rate certain such criteria as not applicable in some cases.
Each of the criteria are at least partially influenced by, and appear in, the following seminal publications:
- Coverage by the News Media of the Benefits and Risks of Medications. Ray Moynihan, B.A., Lisa Bero, Ph.D., Dennis Ross-Degnan, Sc.D., David Henry, M.B., Ch.B., Kirby Lee, M.A., Judy Watkins, B.A., Connie Mah, B.A., and Stephen B. Soumerai, Sc.D.N Engl J Med 2000; 342:1645-1650 June 1, 2000 DOI: 10.1056/NEJM200006013422206
News Story Review Criteria
(news release review criteria appear lower on the page)
About 70% of the stories we review fail to satisfy this criterion.
- Its not good enough to say, “The cost is much lower than the invasive procedures.” What is that cost? How much lower?
- Do insurance policies cover the intervention?
- Might there be costs for copays and other expenses (psychologists, dietitians, etc.) that amount to significant additional expenditure? If you cant quantify them you can at least mention them.
- It may be difficult to estimate costs of an experimental approach early in its development. But can you at least cite costs of existing alternatives? Is the new approach comparable to other approaches whose costs you can cite? Our rule of thumb: If it’s not too early to talk about how well something might work, then its not too early to start discussing what it may cost.
Criterion #2 Does the story adequately quantify the benefits of the treatment/test/product/procedure?
We expect stories to give readers some sense a quantitative estimate – of the scope of the potential benefit. Many stories:
- Use only relative, not absolute, risk reduction or benefit data.
- Rely too heavily on what may be unrepresentative patient anecdotes about benefits.
- Cite statistics appropriately, but allow even a single unchallenged exaggerated quote to throw the story out of balance
Many stories emphasize or exaggerate potential benefits while minimizing or completely ignoring potential harms. They may:
- Fail to mention potential harms.
- Fail to quantify potential harms.
- Fail to describe the severity of potential harms.
- Fail to account for “minor” side effects that could have a significant impact on a patient’s life.
- Rely too heavily on a patient anecdote about safety.
- Rely too heavily on a researchers comment that an approach appears to be safe – without supporting data.
We expect journalists to critically evaluate the evidence, not merely to take published or presented research as gospel. Many stories:
- Fail to point out the limitations of the evidence.
- Fail to include a caution about interpretation of uncontrolled data.
- Fail to point out if a primary outcome is a surrogate marker or fail to caution readers/viewers about extrapolating this to health outcomes.
- Fail to point out the limited peer review that may have taken place with findings presented at a scientific meeting.
- Present anecdotes as evidence of a treatment’s harms or benefits – rather than as a single illustration of its use.
This criterion assesses whether a story exaggerates or over-sells a condition. There are different forms of “mongering” – turning risk factors into diseases (e.g., low bone mineral density becomes osteoporosis); misrepresentation of the natural history and/or severity of a disease (e.g., early-stage low-grade prostate cancer); medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction or female ‘sexual dysfunction’); medicalization of normal states (baldness, wrinkles, shyness, menopause); or exaggeration of the prevalence of a disorder (e.g., using rating scales to ‘diagnose’ erectile dysfunction).
In order to get a satisfactory score on this criterion, two things must occur: There must be an independent expert source quoted (someone not directly connected with the research) and there must be some attempt to let the public know about potential conflicts of interest.
We understand that news wants to cover whats new. But we think theres an obligation to put the new into the context of existing alternatives that have a longer, more proven track record. Unsatisfactory stories may:
- Discuss a surgical approach without mentioning non-surgical alternative approaches.
- Discuss a new test without mentioning other tests that are available, including the option of not being screened in the case of a screening test.
- Fail to discuss the advantages/disadvantages of the new idea compared with existing approaches.
- Fail to discuss how the new treatment, test, product or procedure fits into the realm of existing alternatives.
Does the story make a drug sound like its available at the corner drug store (or may be soon) when its really only in an early-phase trial? Does it feature one doctor promoting a technology without providing a sense of how many trained doctors there are, or what the learning curve is? Did it allow promoters to make unfounded predictions of FDA approval or widespread availability? There needs to be some sense of history of the thousands of previous flashes in the pan in medical research that were equally exciting but never panned out.
Many “new” treatments, tests, products or procedures are not really novel. The agent may be another member of a well-established therapeutic class of drug; even if it represents a new class, it may offer no more than the drugs that are widely available. In the press release for a new drug this essential information may be lost in the hype and the drug is portrayed as “novel” in order to increase initial sales. We believe that journalists should be able to make these distinctions by doing their research or by asking independent sources.
It is simply not sound journalistic practice to lift material directly from a news release without stating that this is the case. This criterion will rarely be employed. If, on the front end, we know that a news release was sent out and if we can get a copy of that release, we will review it along with the story.
- For a story to be rated unsatisfactory, we must have clear evidence that the exact same wording from a news release appears in a story.
- Lacking such evidence, the worst a story can rate is N/A.
- With a story that has several independent sources, we can safely assume that it did NOT rely on a news release and thus, give it a Satisfactory score.
News Release Review Criteria
The 10 criteria used to review news releases are almost identical to those above.
Criterion #6 is slightly different for news release reviews. It gives this guidance:
Does the news release identify funding sources & disclose conflicts of interest?
Potential conflicts of interests should be disclosed in an editor’s note included in the release. The funding for the project must also be disclosed along with any “official” tie investigators may have with that funder.
The other significant difference is Criterion #10. That final criterion for news release reviews is this:
Does the news release include unjustifiable, sensational language, including in the quotes of researchers?
Red flags rise up in our eyes when we see terms such as:
- could become the new standard of care
- Holy Grail
- simple blood test
- this might/may/could lead to . (which means, of course, that might/may/could not applies as well)