Note to our followers: Our nearly 13-year run of daily publication of new content on HealthNewsReview.org came to a close at the end of 2018. Publisher Gary Schwitzer and other contributors may post new articles periodically. But all of the 6,000+ articles we have published contain lessons to help you improve your critical thinking about health care interventions. And those will be still be alive on the site for a couple of years.
Read Original Story

Enthusiasm for “100% accurate” pancreatic cancer test does not reflect limited evidence quality

Rating

2 Star

Categories

Blood Test Might Detect Deadly Pancreatic Cancer in Early Stages

Our Review Summary

This story is about a study of a new test for pancreatic cancer. It covered a lot of ground in a short space. It included many voices and a lot of good background about pancreatic cancer. It missed some big areas, though, including the potential costs of the tests and treatments for the disease, the quantification of the benefits, and any explanation of harms that would come from widespread screening for pancreatic cancer. It was stronger than a competing story from the Houston Chronicle about this study, and similar in quality to a competing LA Times piece.

 

Why This Matters

Pancreatic cancer is a deadly cancer that often presents insidiously. By the time of diagnosis the cancer is often advanced and incurable. There are no current screening tests for detecting early-stage cancer. A tumor marker that can both detect early cancer and monitor treated patients for cure/recurrence would potentially have great value.

Criteria

Does the story adequately discuss the costs of the intervention?

Not Satisfactory

There is no mention of costs in this story. No costs for the test. No costs for the downstream treatments and surgeries. If a researcher is going to be quoted as saying a clinical test could be on the market in a year, there should be some mention of costs.

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The story does not actually quantify the benefits of this screening test. It skirts the issue by saying that the test detected cancer in 100% of the people who were known to have cancer. That description isn’t inaccurate, but it doesn’t fully (or fairly) convey what would be the true value of a screening test: detecting cancer at an early — potentially curable — stage.  The story suggests that the test might have saved the life of Dr. Teresa Flippo-Morton, a prominent breast surgeon from Charlotte, N.C., whom we presume has died of invasive pancreatic cancer (although this is never explicitly stated, and the story later suggests her life could still perhaps be saved). That’s problematic, because while the test identified all of the people known to have cancer, only 5 patients in the study had early stage cancer. (The story states, erroneously it would seem, that there were 7 such patients.) That’s a very small sample from which to draw conclusions about the value of the test. In addition, while patients who are able to undergo early treatment might be cured of the disease, the NCI cancer summary points out that the 5-year survival rate for patients who underwent complete resection for small pancreatic tumors that were confined to the pancreas (the target for screening) was still only 18 to 24%. So, patients may still face long odds even when their tumors are caught early — context the story could have provided.

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

The harm from screening and treatment is an important issue — these are major surgeries (which are best performed by skilled surgeons at high-volume centers) that are associated with mortality rates of around 4% and multiple morbidities. While the test was 100% sensitive and specific in the study, in the real world there will be false negatives and false positives–and possibly indeterminate results. We also do not know whether there is any risk of overdiagnosis (detecting cancers that would never cause a problem).

But there was no mention in the story of the potential harms. The logical extension of the approach advocated in this piece is that all people should be screened for pancreatic cancer. If that is what the researchers and the other commentators in the piece believe, then they should also address the issue of unnecessary surgeries, side effects from treatments, and the anxieties caused by mass screening. A more reasonable point of discussion in the piece might have been to suggest developing a screening protocol for populations known to be at risk. For example, people who smoke, are obese, or who have diabetes are all more likely to be diagnosed with pancreatic cancer. An effective and cost-effective screening program would need to determine which people to target (including whether a person is healthy enough to undergo surgery) and at what age.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

The story earns credit for including skeptical comments from Dr. Wolpin about the limited applicability of the results. He notes that “In this study the patients were known to have cancer or not to have cancer. In this kind of sample sensitivity and specificity tend to look good.”

But while those comments are on-target, they come at the very end of the story and feel bit like an afterthought to the strong language throughout the rest of the piece, essentially heralding this test as a possible cure for cancer. The very first sentence says, “With just a drop of blood doctors may one day be able to detect pancreatic cancer in its early stages, before it has become deadly, a new study suggests.” Then later it says, “But, by good fortune, the protein turns up in exosomes only when there is cancer, so its presence could be an early, and testable, marker for the disease.” The story even suggests that the test could have saved the life of a prominent surgeon. Such enthusiasm is a bit out of balance with the quality of evidence on display here, some of which was based on mouse models. This test is far from ready for use in clinical practice.

Does the story commit disease-mongering?

Satisfactory

We wish more stories took the time to explain to readers how common or rare a condition is. This story gets high marks for pausing to say, “For perspective, over the course of a lifetime, 1.5 percent of Americans will develop pancreatic cancer.”

Does the story use independent sources and identify conflicts of interest?

Satisfactory

We give the story a pass here because there are multiple voices in the piece. But it should be noted that there was only one dissenting opinion, at the very end, from Dr. Brian Wolpin, of the Dana Farber Cancer Institute. He brought up the point that should have been in the second paragraph of the story: “In this study the patients were known to have cancer or not to have cancer. In this kind of sample sensitivity and specificity tend to look good.”

Does the story compare the new approach with existing alternatives?

Not Satisfactory

There is no mention of alternatives in this story or whether other research has gone in this direction and failed to yield a marketable test. While there really aren’t any good screening tests, there is another tumor marker (CA 19-9) that has been used for screening, but it does not readily detect the early-stage cancers. The study actually compared the new test with CA 19-9.

Does the story establish the availability of the treatment/test/product/procedure?

Satisfactory

We were surprised to see mention in the story that a clinical test could be available within a year. Given that the test will need to be further validated in a much larger group of study subjects, we think that’s very unlikely. The story does frame the comment as “just speculation” and notes that additional research will be needed. We’ll give the benefit of the doubt.

Does the story establish the true novelty of the approach?

Not Satisfactory

The findings are presented as novel, but novelty is not established. There’s no discussion of whether there has been other research into tests for pancreatic cancer and how the new test builds upon those previous efforts. We can infer that this is new, but the context is not established.

Does the story appear to rely solely or largely on a news release?

Satisfactory

The story does not rely on a news release.

Total Score: 4 of 10 Satisfactory

Comments (5)

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.

Marc Beishon

June 26, 2015 at 5:20 pm

You’re doing great work here but could you also link to the study paper or abstract? The news stories often don’t do that either and it’s frustrating to spend time looking for it although in this case it was easy to find: http://www.nature.com/nature/journal/vaop/ncurrent/full/nature14581.html

Reply

    Kevin Lomangino

    June 27, 2015 at 8:46 am

    Thanks for the reminder. We’ve added a link in the review summary.

    Reply

      Marc Beishon

      June 27, 2015 at 1:00 pm

      I think something that would improve all this would be if researchers were forced to write a lay abstract as well as a scientific one. A lay description is required in a lot of grant applications now and some journals I believe.

      Gary Schwitzer

      June 27, 2015 at 4:43 pm

      Marc,

      Thanks for your comment.

      Lay abstracts may be a worthwhile idea in principle. But the execution of the idea could be fraught with even worse problems. If a scientific summary can be skewed to make a finding look more impressive, what might happen with liberties taken in a lay abstract? It seems that journal editors have a difficult enough time editing the scientific summaries that are submitted. Editing and verifying the veracity of lay summaries adds to that burden. Indeed, if journal news releases can be viewed as somewhat akin to lay summaries, we have seen enough problems with those.

      On the other hand, when done well, lay summaries can be very helpful for all parties involved. I would not be so quick to say that they “would improve all of this” but that they “could” – with caveats.

      Marc Beishon

      June 28, 2015 at 5:29 am

      Gary,
      I think there is value here because in my experience study authors are pretty good at explaining the real world significance of their findings and if they write a lay abstract themselves that would help the ‘lost in translation’ we see when the PR people and journalists start on it. They could use a guideline that takes your criteria for judging news stories and so translate statistical significance into relative gains, and note costs and limitations.

      In this pancreatic cancer example, the abstract ends with:

      “crExos may serve as a potential non-invasive diagnostic and screening tool to detect early stages of pancreatic cancer to facilitate possible curative surgical therapy.”

      I don’t think it’s too much to ask them to translate the ‘may’, ‘potential’ and ‘possible’ into a few lines that state clearly the limitations that this hints at but are obviously known in the research community.

      My son, who is at med school here in England, wrote an assessed research paper last year in which he had to write both scientific and lay abstracts – so hopefully this is a sign of things to come!