May 09 2008
|
home >> ratings information
rating information
News stories will be monitored each Monday through Friday in major U.S. media. We will monitor and select from the following:
- the nation's top 50 circulation newspapers as ranked by the Audit Bureau of Circulation (10 newspapers per day, drawing one from #1-5, one from #6-10, etc.);
- The evening network newscasts of ABC, CBS and NBC;
- The Associated Press wire service;
- The weekly news magazines TIME, Newsweek and U.S. News & World Report
(As time permits, we will also review stories sent to us by users – stories from other media, or stories that our cyclical reviews might miss.)
News stories will be identified by daily reviews of the major media outlet websites using a hand-searching approach.
In order to be eligible for review, the article should:
- Be relevant to the management and prevention of disease and in particular should relate to claims about treatments, procedures or tests.
- Discuss an intervention that is intended for use in humans or one for which the link to human application is at least implied.
- Make explicit or implicit claims about the efficacy of treatments, procedures or tests
review criteria
| Rating Criteria |
Not Satisfactory |
Satisfactory |
| Novelty of Treatment |
Does not mention (or inaccurately represents) if treatment or test is genuinely new or just a new twist on an older idea (in the case of drugs, a re-formulation of an existing treatment, or another member of a well established drug class). |
Accurate information on novelty (or lack thereof) |
| Availability of Treatment |
Does not mention availability of treatment or test in U.S. (and/or, in the case of drugs, treats FDA approval as a fait accompli). Fails to justify a stated projection of future availability and/or fails to describe the basis and reliability of the source making that projection. |
Accurate information on availability of treatment in U.S. If the story projects possible future availability, it justifies the projected time frame and describes the basis and reliability of the source making that projection. |
| Treatment Options |
No mention of alternatives or no mention of possible advantages/disadvantages of the new idea compared with existing approaches. |
Mentions appropriate alternatives and provides advantages/disadvantages of the new idea compared with existing approaches. Explains how the new treatment (or test) fits into the realm of existing approaches. |
| Disease Mongering |
Frames risk factors (e.g. high cholesterol) as a disease, or fails to mention (or misrepresents) natural history of disease, or exaggerates prevalence or incidence, or medicalizes a normal human variation. |
No obvious elements of disease-mongering |
| Quality of Evidence |
No mention of the nature of clinical evidence, especially the importance of randomized clinical trials.
Mention of the nature of the evidence but interpretation or discussion is incomplete and not helpful. |
Where relevant, there is mention of strength of evidence and correct interpretation, demonstrating a grasp of the hierarchy of evidence. (e.g., a story about a non-randomized cohort or observational study should explain that researchers were not able to adjust for all factors that might explain an observed difference.) |
| Quantifying treatment benefits |
No quantitative estimate of benefit
Quantitative estimate in relative frame only |
Estimate in both absolute and relative frames, or absolute frame only, or rates with & without treatment |
| Treatment Harms |
No mention of harms, or discounts potential harms; minimizing potential harms. |
Balanced information about harms (frequency or seriousness) |
| Treatment Costs |
No mention of costs, or downplays cost as an issue; or mentions cost only, but fails to present comparative cost information with other treatments (or tests) for the same condition. |
Mentions comparative costs and comments on cost-effectiveness. |
| Sources of Information |
No mention of sources or possible conflicts of interest;
No attempt at independent corroboration, relying on a single source. |
Provides detail on information sources and their potential conflict of interest, and reports independent source, or mentions unsuccessful attempt to obtain corroboration |
| Relies on Press Release* |
Evidence from news release or other news stories that the journalist has relied on a press release as the main information source and used the text in the story |
Did not appear to rely largely or solely on news release. |
| * Because of difficulty in gathering news releases for the number of stories we're monitoring, we are not likely to use this criterion except in instances where the news release is readily available. |
Scoring:
For each news article, the 10 criteria will be scored as "satisfactory", "unsatisfactory" or "not applicable". Total scores are posted for articles that have two or fewer "not applicable" ratings, and are expressed as proportions. For the purposes of display, the total scores are translated into a star rating.
| Percent score assigned |
"Star" score assigned |
| 0% |
0 stars |
| 1-20% |
1 star |
| 21-40% |
2 stars |
| 41-60% |
3 stars |
| 61-80% |
4 stars |
| 81-100% |
5 stars |
(If one or two items are rated 'NA', the denominator will be 8 or 9 rather than 10.)
|
