What does an 87% accurate Alzheimer’s test mean? Not much without positive predictive value.

What does it mean to say that a new test is 87% accurate?

I think most journalists writing about a paper published in the journal Alzheimer’s & Dementia could not satisfactorily answer that question – about a statistic they repeatedly quoted in stories about that paper.

The answer is:  Not much, if you don’t take positive predictive value into account.

And again, most journalists writing about the study won’t know what this means either.

The BBC called it “a major step forward in developing a blood test to predict the onset of Alzheimer’s disease.”

I’ve often warned readers to head for the hills if anyone promises a “simple blood test.”  Are you in the hills yet? Because Reuters reported, “Study paves way for simple blood test to predict Alzheimer’s.” While there were caveats at the very end of the story, who at Reuters is responsible for this simplistic headline?

Other stories called it “a major breakthrough.”  One stated definitively that “A simple blood test can predict Alzheimer’s Disease” while another said “Alzheimer’s blood test not far away.” I’m not sure that either news story should make the claim it made at this point in time.

NHS Choices “Behind the Headlines” analysis

Now let’s get to the analysis of the claims.

The NHS Choices Behind the Headlines site wrote, “Blood test for Alzheimer’s ‘no better than coin toss.‘ ” Excerpt:

The media coverage was broadly accurate, but none reported the positive predictive value of the test. This reduces the impressive sounding 87% accurate figure to around the 50% level, depending on the prevalence assumptions, giving the test the same predictive value as a coin toss.

This important information should have been highlighted to avoid overstating the utility of the test on its own.

In a HypeWatch column on MedPage Today, John Gever writes, “Another Dementia Blood Test Oversold.” Excerpts:

Only about 10% of patients of patients with MCI convert to clinical dementia per year. With nearly 30% of positive results false (remember, the specificity was 71%) as well as 15% of negative results false, most of the positive results in such a group will be false.

Yes, it’s time once again for a tutorial in positive predictive values. If we have 100 MCI patients and a 10% conversion rate, then 10 of them will develop dementia. These are the true positives. There will be 90 true negatives — the ones who don’t convert.

But with a specificity of 71%, the test will falsely identify 29% of the 90 true negatives, or 26, as positive. Meanwhile, with a false negative rate of 15%, only nine (rounding up from 8.5) of the 10 true positives will be correctly identified.

That’s 26 false positive results against nine correctly positive. That’s useless in a clinical setting. In fact, it’s worse than useless, since the false-negative results will expose patients to unnecessary clinic visits and treatments, and generate anxiety for them and their families.

But here’s lead author Abdul Hye from King’s College London in the press release: “We now have a set of 10 proteins that can predict whether someone with early symptoms of memory loss, or mild cognitive impairment, will develop Alzheimer’s disease within a year, with a high level of accuracy.”

…

In the U.S., most coverage (such as this at Huffington Post) appeared to be simply rewrites of the press release with no independent viewpoint at all.

Sigh.

We’ve been down this road before.

And we shall travel it again.

————————–
Tweet

Follow us on Twitter:

https://twitter.com/garyschwitzer

https://twitter.com/healthnewsrevu

and on Facebook.