NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine - https://archive.org/web/.
Read Original Story

Breath test may be able to detect common cancers

Rating

1 Star

Breath test may be able to detect common cancers

Our Review Summary

If you’re going to report on preliminary research, you have to be clear that it’s preliminary. This was a laboratory study, at best years away from practice.

 

Why This Matters

Our own "electronic nose" detected an odor that this research is extremely preliminary, especially after pulling up the study itself. There is in fact no clinical test, and this lab study was about the development of a test. It didn’t assess the technique’s purported ability to detect cancer earlier, better, more reliably, or more efficiently than what is done now. The forward-looking statements are speculative and unchallenged.

The story also glosses over some big, sticky issues that come up in early detection, like the "subjectively healthy" controls, the uncertain continuum between early cancerous signals ("cancer risk") and actual clinically significant cancers, and the early detection and treatment of "pre-" prostate and breast cancers. If we’re going to pump this hypothetical tool’s use in early detection, we really need to study its ability and consequences in identifying the different biological entities, of varying clinical significance, on the odd family tree we collectively call cancer.

This is the story of an idea, a hypothesis that is ready to be studied, not a sea change in how we detect cancer. If we’re lucky, it will lead to development of a new technique that proves better than the ones we currently employ, but at best we’re years away from having it in a hospital.

See the Daily Mail’s coverage of the study for many more of the nuances, caveats, and appropriate tone that we asked for in this review — some of which were provided by Dr. Kuten himself, e.g: "We need to carry out a lot more work on many more people in different stages of the cancer. We also need to find out the extent to which it can diagnose cancer. This will probably take at least two years. It will be several years at least before it is available for use."

Criteria

Does the story adequately discuss the costs of the intervention?

Not Satisfactory
Costs aren’t known, so it’s too early to evaluate cost-effectiveness of a tool that hasn’t been developed, without information on sensitivity, specificity, and how many unnecessary follow-up tests and treatments are prompted because of false positives when used during early detection.

But for precisely those reasons, the story gets a strike for passing on the unchallenged speculations that the hypothetical tool could be cheap.  Unsound journalistic practice.

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory
While it’s too early to quantify benefits, since this wasn’t a clinical study, we think the story is misleading in extrapolating the potential benefits, or paraphrasing the researchers’ hopes for their research’s success without question. It says "the team found they could not only distinguish between healthy and malignant breath but also identify the four different common tumor types." It was a retrospective analysis of data, not a blinded study: these were not clinical distinctions or clinical identifications of people with cancer and those without. As we’ve noted, the "healthy people" were defined by subjective descriptions while those with cancer had been diagnosed using conventional tests. We think the benefits described are misleading.

Does the story adequately explain/quantify the harms of the intervention?

Not Applicable
There’s no tool yet so it’s too early to evaluate the potential harms of poor sensitivity (how many cancers pass by its nose undetected) or poor specificity (how many people without cancer are flagged, biopsied, etc). 
 
It would’ve been nice if the article alluded to the fact that not all diagnostic and screening tools are created equally, and clinical studies are needed to determine whether a test will cause more harm than good, or how that trade-off compares to our current approaches.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

There’s almost no evaluation of the evidence. As we discussed in the "Availability" criterion above, this was a preliminary, retrospective, statistical analysis of data in a laboratory, not a clinical test of an actual diagnostic technique. Readers should’ve been told the difference. 

Citing 177 people is a big number but a meaningless one without knowing what was done. (A hyperbolic example: consider the quality of evidence from a web poll with 177 respondents compared to a randomized, controlled, double-blind, multi-center clinical study in 177 people.) We’re told vaguely that more work is needed, but the sentence construction — "While more work is needed, the early success…" — makes clinical success seem like a foregone conclusion. People would be shocked to learn how many techniques with "early success" ultimately fail in the real world where real lives are at stake. The researcher states "if we can confirm these initial results in large-scale studies," which is good of him, but the tone and unbounded optimism of the article gives us no reason whatsoever to think this preliminary evidence won’t be confirmed.

The evidence in this lab study, in fact, appears to have used data from existing laboratory techniques of gas chromatography/mass spectrometry to inform that of the novel nanoarray, and also compared the uses of the techniques. It highlights the early nature of the evidence, as the study was about the development and laboratory analysis of the technique, not clinical outcomes.

There’s another limitation in the evidence that’s worth pointing out. According to the abstract: "The healthy population was healthy according to subjective patient’s data." The study itself states that the cancer population had been diagnosed by conventional tests. That questions the relevance of the study to the article’s presentation of the approach as a way to detect cancer early — before it has symptoms, before it would be diagnosed otherwise. Because we don’t know if the control group also had early but asymptomatic or undiagnosed cancer.

Should a reporter be expected to critically evaluate all these questions when writing a 6 paragraph summary of a study? We think that if the study is as preliminary as this one, and the technique as far from touching clinical practice as this one, then yes, we expect an article to do more than repeat the optimistic hopes of the investigators themselves without any independent evaluation. Otherwise, isn’t it just a press release from the investigators?

Does the story commit disease-mongering?

Not Satisfactory

The story is vague and speculative about the advantages of the theoretical clinical test, such as the presumed uses and benefits in early detection. By being vague, it casts a wide net over an undefined population that the technology could presumably help. The story mentions the "four different common tumor types" of lung, breast, bowel and prostate.  But not all cancers are equal.  Not all pre-cancers are equal.  There is controversy even with today’s screening methods about the benefits/harms of screening for some of these cancers in some populations.  None of that nuance is even touched on.  Should readers be worried about EVERYTHING? That’s disease-mongering.

Does the story use independent sources and identify conflicts of interest?

Not Satisfactory
No independent sources were used to balance the potential promises with any of the caveats we’ve provided in this review.
The affiliation of the one quoted researcher is identified. The researchers themselves did not disclose any relevant conflicts of interest in the study.

Does the story compare the new approach with existing alternatives?

Not Satisfactory
The story delivers no conception of the technique’s potential role in the current landscape of health care, nor what need it could fill. The published study itself describes the need that motivated the development of this breath test: the current screening techniques are either unreliable, invasive, associated with their own harms, and/or not useful in all populations. (Note that this list of reasons doesn’t align with most of the potential benefits of the technique described by the article.)

Even without detailing how the hypothetical tool might compare to existing alternatives, or fill a need, the story at least could’ve told us that the reason further testing is needed is not just to repeat the research on a larger scale but also in order TO COMPARE the approach with what we’re currently doing.

Does the story establish the availability of the treatment/test/product/procedure?

Not Satisfactory
It’s clear the tool is not in clinical practice yet, but its stage of development is muddled. Without evaluating the quality of evidence, readers can’t tell the difference between this preliminary laboratory research and an actual clinical tool, to say nothing of an actual clinical study to test whether it’s better/worse than what we currently use.

Reuters posted just one reader comment before closing the comment section.  It read as follows – typos and all: "This is a terrific Leap in Teachnology and should be made available Worldwide at hospitals, clinics and Good Dr. Offices." Readers should not have been led to believe the simple, proven breath test is at hand, ready to deploy to doctors. It’s years away if we’re lucky.

Does the story establish the true novelty of the approach?

Satisfactory

On the plus side, it tells us that the study builds on earlier research from last year, and we give it the benefit of the doubt. On the minus, the story is vague about the novelty of breath testing for cancer, which was not a new approach invented by this group in even its last study. 

 

Does the story appear to rely solely or largely on a news release?

Not Applicable
We haven’t found a news release. However, we are curious what the sources were since none were stated and another article on the study has the same exact quote from Dr. Kuten.

Total Score: 1 of 8 Satisfactory

Comments

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.