“More accurate than traditional biopsies” and other misinformation about a new prostate cancer test from NY-Presbyterian Hospital

The following is a guest post by Dr. Richard Hoffman, one of our longtime story reviewers and blog contributors. He recently became the Director of the Division of General Internal Medicine for the University of Iowa Carver College of Medicine/Iowa City VA Medical Center.

Last week the American College of Physicians published position papers on achieving high value from cancer screening (see here and here). Ideally, “high-value strategies” deliver a large health benefit relative to harms and costs. Prostate cancer screening, the papers implied, has not been meeting this standard.

One major problem with prostate cancer screening is that the prostate-specific antigen (PSA) test is not very accurate. Only about 30% of men with an abnormal PSA result will have prostate cancer—as determined by a prostate biopsy. This low predictive value has led investigators to evaluate strategies for reducing unnecessary biopsies. While these strategies theoretically could reduce false positives, studies often failed to fully address the problem of overdiagnosis—finding slow-growing, low-grade tumors that would never cause problems during a man’s lifetime. Overdiagnosis and resulting overtreatment undermine the value of prostate cancer screening.

A study recently presented at the American Urological Association’s annual meeting addressed the overdiagnosis issue. Investigators showed that a novel gene signature test performed on a urine specimen could better predict which men with an intermediate-level elevation of PSA (2 to 10 ng/mL) would have a high-grade prostate cancer (Gleason scores of 7 or greater) on biopsy. Based on the study results, investigators postulated that the new test could reduce biopsies by 27%, while missing few of the highest-grade tumors.

These findings are interesting but preliminary. The study was not designed to address whether using this test for clinical decision making actually improved patient outcomes—fewer biopsies and at least similar cancer control—compared with other ways of assessing risk.

Unfortunately, New York-Presbyterian Hospital, home of the lead investigator, issued a news release that was notable for misinforming readers about the study. The release was titled, “Urine test detects prostate cancer with 92 percent accuracy, according to study.”  The AUA abstract presented enough data for me to construct a 2 x 2 table.

The only “92%” number calculable from the data is the sensitivity—the proportion of high-grade cancers that can be detected with the new test (137/149). But the news release talks about “accuracy,” which as defined in the epidemiology literature is a fairly useless concept. It refers to the proportion of all tested subjects who are correctly classified—the true positives + the true negatives.  In this case, the “accuracy” of the new test is only (137+126)/519 = 51%.  Although not mentioned in the press release, the abstract reported the area under the ROC curve (AUROC)—which is the best overall measure of how well a test distinguishes between people with disease and those without. The AUROC for the gene test was 0.72 (a value of 0.5 corresponds to a worthless test while 1.0 corresponds to a test with perfect accuracy)–indicating fair to good performance.

I believe the most important implications of the study are that if men with a negative gene test were not biopsied, then overall 27% (138/519) fewer men would undergo biopsy. These men would avoid the anxiety and risk of complications — including potentially serious infections — that accompany this procedure. Furthermore, among these men avoiding biopsy, 91% (126/138; the negative predictive value) would not have a high-grade cancer—but 9% would. Men who are making biopsy decisions based on the new test would need to understand this tradeoff.

But the release does not frame the results in a way that would promote such informed decision-making. In fact, some of the quotations in the news release were downright perplexing. For example:

• The release quotes the study author speculating that the new test could “replace traditional surgical biopsies.” But that’s not what the study suggests. Biopsies will still be necessary to confirm prostate cancer in those who test positive; the study shows potential just to reduce the number of unnecessary biopsies.
• The release says the test “should reduce” biopsies by 27%, but “might reduce” would be more accurate. The carefully selected study group might not be a representative population, and the test might not work as well in a bigger, more diverse group.
• Finally, the study author states in the release that the new test is “more accurate than traditional biopsies.” But biopsies, although imperfect, remain the gold standard for screening tests. The new test is not “more accurate.” It just better predicts the presence of high-grade disease compared to PSA.

What could this news release have done differently to more effectively communicate the value of this research? In my view, highlighting a meaningless “accuracy” statistic and presenting misleading quotes about the new test “replac[ing] traditional surgical biopsies” and being “more accurate than traditional biopsies” just obscure the actual study findings. The release should have focused on the test’s potential to avoid unnecessary biopsies while still identifying almost all of the more aggressive cancers.  A tumor marker that could potentially prevent unnecessary biopsies and reduce overdiagnosis would improve the value of prostate cancer—this is the important message.

[Editor’s note: Contributor Trudy Lieberman recently wrote about her concerns with aspects of New York Presbyterian’s consumer advertising in her post, “Patient anecdotes in NY-Presbyterian hospital ads mask mediocre quality ratings.”]