Review Criteria

Criterion #1 Does the story adequately discuss the costs of the intervention?

About 70% of the stories we review fail to satisfy this criterion.

• It’s not good enough to say, “The cost is much lower than the invasive procedures.” What is that cost? How much lower?
• Do insurance policies cover the intervention?
• Might there be costs for copays and other expenses (psychologists, dietitians, etc.) that amount to significant additional expenditure? If you can’t quantify them you can at least mention them.
• It may be difficult to estimate costs of an experimental approach early in its development. But can you at least cite costs of existing alternatives? Is the new approach comparable to other approaches whose costs you can cite? Our rule of thumb: If it’s not too early to talk about how well something might work, then it’s not too early to start discussing what it may cost.

Read More…

Criterion #2 Does the story adequately quantify the benefits of the treatment/test/product/procedure?

We expect stories to give readers some sense – a quantitative estimate – of the scope of the potential benefit. Many stories:

• Use only relative, not absolute, risk reduction or benefit data.
• Rely too heavily on what may be unrepresentative patient anecdotes about benefits.
• Cite statistics appropriately, but allow even a single unchallenged exaggerated quote to throw the story out of balance

Read More…

Criterion #3 Does the story adequately explain/quantify the harms of the intervention?

Many stories emphasize or exaggerate potential benefits while minimizing or completely ignoring potential harms. They may:

• Fail to mention potential harms.
• Fail to quantify potential harms.
• Fail to describe the severity of potential harms.
• Fail to account for “minor” side effects that could have a significant impact on a patient’s life.
• Rely too heavily on a patient anecdote about safety.
• Rely too heavily on a researcher’s comment that an approach appears to be safe – without supporting data.

Read More…

Criterion #4 Does the story seem to grasp the quality of the evidence?

We expect journalists to critically evaluate the evidence, not merely to take published or presented research as gospel. Many stories:

• Fail to point out the limitations of the evidence.
• Fail to include a caution about interpretation of uncontrolled data.
• Fail to point out if a primary outcome is a surrogate marker or fail to caution readers/viewers about extrapolating this to health outcomes.
• Fail to point out the limited peer review that may have taken place with findings presented at a scientific meeting.
• Present anecdotes as evidence of a treatment’s harms or benefits – rather than as a single illustration of its use.

Read More…

Criterion #5 Does the story commit disease-mongering?

This criterion assesses whether a story exaggerates or over-sells a condition. There are different forms of “mongering” – turning risk factors into diseases (e.g., low bone mineral density becomes osteoporosis); misrepresentation of the natural history and/or severity of a disease (e.g., early-stage low-grade prostate cancer); medicalization of minor or transient variations in function (e.g. temporary erectile dysfunction or female ‘sexual dysfunction’); medicalization of normal states (baldness, wrinkles, shyness, menopause); or exaggeration of the prevalence of a disorder (e.g., using rating scales to ‘diagnose’ erectile dysfunction).

Read More…

Criterion #6 Does the story use independent sources and identify conflicts of interest?

In order to get a satisfactory score on this criterion, two things must occur: There must be an independent expert source quoted (someone not directly connected with the research) and there must be some attempt to let the public know about potential conflicts of interest.

Read More…

Criterion #7 Does the story compare the new approach with existing alternatives?

We understand that news wants to cover what’s new. But we think there’s an obligation to put the new into the context of existing alternatives that have a longer, more proven track record. Unsatisfactory stories may:

• Discuss a surgical approach without mentioning non-surgical alternative approaches.
• Discuss a new test without mentioning other tests that are available, including the option of not being screened in the case of a screening test.
• Fail to discuss the advantages/disadvantages of the new idea compared with existing approaches.
• Fail to discuss how the new treatment, test, product or procedure fits into the realm of existing alternatives.

Read More…

Criterion #8 Does the story establish the availability of the treatment/test/product/procedure?

Does the story make a drug sound like it’s available at the corner drug store (or may be soon) when it’s really only in an early-phase trial? Does it feature one doctor promoting a technology without providing a sense of how many trained doctors there are, or what the learning curve is? Did it allow promoters to make unfounded predictions of FDA approval or widespread availability? There needs to be some sense of history – of the thousands of previous flashes in the pan in medical research that were equally exciting but never panned out.

Read More…

Criterion #9 Does the story establish the true novelty of the approach?

Many “new” treatments, tests, products or procedures are not really novel. The agent may be another member of a well-established therapeutic class of drug; even if it represents a new class, it may offer no more than the drugs that are widely available. In the press release for a new drug this essential information may be lost in the hype and the drug is portrayed as “novel” in order to increase initial sales. We believe that journalists should be able to make these distinctions by doing their research or by asking independent sources.

Read More…

Criterion #10 Does the story appear to rely solely or largely on a news release?

It is simply not sound journalistic practice to lift material directly from a news release without stating that this is the case. This criterion will rarely be employed. If, on the front end, we know that a news release was sent out and if we can get a copy of that release, we will review it along with the story.

• For a story to be rated unsatisfactory, we must have clear evidence that the exact same wording from a news release appears in a story.
• Lacking such evidence, the worst a story can rate is N/A.
• With a story that has several independent sources, we can safely assume that it did NOT rely on a news release and thus, give it a Satisfactory score.

Read More…