Last week many news organizations reported on research to find a blood test to help identify Alzheimer’s disease.  We reviewed four stories on that research:

• San Francisco Chronicle
• ABC’s Good Morning America 
• ABC Evening News
• CBS Evening News 

All four stories were rated as unsatisfactory for discussing the quality of the evidence.
Three out of four were unsatisfactory in discussing potential benefits and potential harms, in their use of sources, and in their discussion about other existing options beyond this new test.   

Here, specifically, is some of what our reviewers commented on about how the stories could have been better.

How well did the story discuss the quality of the evidence?

San Francisco Chronicle:  Mixed views on this criterion.  The story did a good job in providing actual numbers (e.g., test was positive for Alzheimer's disease in 38 out of 42 patients....). However, the story doesn't give much discussion about the specific limitations of this study. For example, this case-control type of patient selection almost always overestimates test accuracy.  So it's likely to perform worse in real world populations.

Good Morning America: Mixed review.  The segment does a good job, particularly in a broadcast setting, of describing how the research was conducted. However, the story didn't discuss a key limitation: the size of the study group. And it didn't discuss another specific limitation of this study: this case-control type of patient selection almost always overestimates test accuracy.  

ABC Evening News:  The story says that larger studies will be needed to confirm this one, pointing out an important limitation of the study. But the story doesn’t discuss other important limitations of the new research. For example, to judge the accuracy of a new diagnostic screen, researchers must compare it against a “gold standard”. The gold standard in Alzheimer’s requires the examination of brain tissue at autopsy. How can the researchers be sure the people deemed to have Alzheimer’s actually have the disease? If they can’t, then the predictive value of the test is uncertain.  The segment also doesn’t mention that the results come from a case-control study with a design that is inherently susceptible to bias, including the selection of the comparison group, or controls.

CBS:  They failed to describe the type of population that was studied, how representative they may or may not have been. In addition, the story stated this was a study on 259 blood samples.  Rather the test was developed from these samples and tested on a rarified sample of about 50.

How well did the stories address the potential harms from the test?

Reviewers commented that one story or another:

• failed to address the potential emotional and financial toll of a diagnosis for the patient and the patient’s family and the insurance ramifications.
• failed to address the possibility of, or rate of, false positive test results

How well did the stories address the potential benefits from the test?

Reviewers commented that one story or another:

• failed to make it clear what one would do with advance knowledge that one was going to develop Alzheimer's disease.
• failed to explain that what sounds like an extraordinarily powerful result – a 90% accurate test – came from a very small sample size.
• stated that the test identified Alzheimer’s in 90% of cases (a measure of the test’s “sensitivity”). But it doesn’t state the “specificity” of the test—which includes a measure of false-positive results.
• didn’t discuss whether the researchers compared the new test against the diagnostic gold standard—analysis of brain tissue at autopsy.
• presented a tone that knowing about the disease early would allow for treatment to benefit the patient when this is not the case given the lack of available proven treatments.

How well did the story seek independent sources?

Reviewers commented that one story or another:

• failed to disclose some obvious conflicts of interest. Four of the authors have commercial ties to the company that developed the test.
• included person-on-the-street interviews that didn’t add anything. It's clear none of these people was well informed enough to have a meaningful opinion.
• quoted someone from the company developing the test and quoted only one "expert" but he did not comment on the test but only on the subtlety of the disease.

Three of four stories did not address costs and that’s understandable given the early stage of research.  But ABC quoted an investigator who said that the test is “cheap” and “inexpensive”. The story did not cite a dollar amount, nor did it compare the cost of the new test to that of current testing. Even if the cost of an individual test can be predicted, there will doubtless be high additional "downstream" costs associated with satisfying the curiosity of all the people at low risk of Alzheimer’s requesting low-yield blood tests. So the “cheap” and “inexpensive” comment may not give a complete picture of true costs.

We hope you find these cross-media comparisons helpful.  For us, it provides a re-affirmation that our evaluation criteria can be important reminders for journalists about the kind of information readers and viewers need in such stories.

No Monday morning quarterbacking here. Just the usual reminder to journalists and consumers that - while in the opera, it ain't over till the fat lady sings - in health care and medical research, exuberance over new ideas such as new drugs maybe ought to be restrained until long-term evidence is in.

Pfizer's announcement this week that it would shelve the insulin-inhaler drug Exubera and take a $2.8 billion pretax hit on the product is a real world reminder. The company faced a far weaker sales market than anticipated, perhaps much of it due to doctors' safety concerns about the drug.

Back when the drug was approved in January 2006, on our HealthNewsReview.org website, we reviewed an AP story about the drug's approval and noted about that story:

"Overall, there is little quantitative information on the efficacy of the new form of insulin compared with the existing method of insulin delivery. While investors may be interested that this could be a "$1 billion a year seller for Pfizer", diabetics who are considering inhaled insulin would appreciate quantitative information on the risks and benefits of Exubera."

I hope journalists are paying attention.

The Chicago Tribune today has a good story about how annual medical checkups are overrated. As the story explains:

That's because there is scant scientific evidence showing that yearly checkups help prevent disease, death or disability for adults with no symptoms. Many tests and procedures performed during the visits have questionable value, experts say.

When many journalists continue to push checkups and screening tests in the absence of good evidence, it is refreshing to see an evidence-based story like this.

On this blog I have been remiss not to weigh in on the recent and ongoing debate over children's health insurance and the SCHIP or State Children's Health Insurance Program.

Not surprisingly, some of the wittiest commentary has come from Jon Stewart and Stephen Colbert. Their segments within the past week should not be missed.

Stewart's coverage on the Daily Show includes a biting commentary from John Oliver.

On the Colbert Report, the host said:

"If we really care for our kids, we should deny them health insurance now to immunize them against expecting it as adults. If we don’t, when they grow up, who knows what other unrealistic things they’re going to expect? You know, if we fund Head Start now, later, they’ll expect education. If we fund school lunches now, later, they’ll expect food."

Who spent more time exploring the issues and looking at the debate: Comedy Central? Or ABC, CBS, NBC, CNN, MSNBC, Fox?