Strengths: Immediately, in the opening words, the story acknowledges that there is another DNA test in development that was reported on this week. It was the only one of the four stories we reviewed on this topic that made this clear.
Weaknesses: It didn’t provide a meaningful comparison of the new approaches with existing colon cancer screening tests.
It also has problems with how the underlying question is presented. Two examples:
This topic is really about when we should pay attention to evidence about a new diagnostic test and how we should evaluate diagnostic tests in comparison with one another- a topic that is often inadequately discussed in both medicine and journalism.
The story says, “Both tests would be less expensive than colonoscopy, and potentially more effective.” No specific cost estimate is given. The Philadelphia Inquirer said “Though the cost has not been set, it could be as low as $300 per test.” Reuters reported that the company said earlier that it would aim for $300-400 per test.
The story also states that “Compliance with colonoscopy is low, since people don’t want to have one, and the overall cost per detection is high” – implying that the new test might be more cost-effective. But it’s unclear at this point if that would be true. For example, previous stool DNA tests have performed much worse (higher costs, lower efficacy) than colonoscopy in modeling studies.
The story did a decent job comparing data presented on two new competing tests. It does not, however, compare the benefits to existing colon cancer screening tests. We will dock points for this in our final criterion on “comparing the new approach with existing alternatives” but will give it the benefit of the doubt on this one.
The two tests the Times reported on were done on blood and on stool, which, in theory, should not harm anyone during the screening. But there is potential for harms as a result of screening. The story states that with the Exact Sciences DNA test “12 percent of the time the patient will be given a false alarm.” And, regarding the Epigenomics blood test, the story states that “more false positives could be generated so you could end up doing more harm than good.”
Mixed bag. The story did provide some important context about test. For example, it says, “Exact Sciences reported in July that its test was highly sensitive and specific when applied directly to cells taken from tumors. But in the real world, the tumor DNA must be detected in stool samples, even though almost all the DNA comes from the bacteria of the gut. Just 0.01 percent of the DNA is human, and most of this is normal DNA, not the altered DNA of tumors.”
But only in passing does it mention that “in the real world, the tumor DNA must be detected in stool samples.” It could/should have driven home the point more clearly that the methodology used in the study tends to overestimate performance compared with an evaluation done in a representative sample population (hence the need to do another study). WebMD, for example, did a better job on this point, with a quote from Dr. Durado Brooks of the American Cancer Society: “This was a very preliminary study that utilized samples from individuals who were known to have colon polyps or cancer. The performance may decrease considerably when the test is used in a large population of healthy individuals.” Health Day quoted Brooks: “Showing that in a small group of samples is very different from demonstrating that in a population where only a small number of individuals are going to have polyps of that size. Then we will know if this is a big step forward.”
There was also no discussion of the limitations of drawing conclusions from an abstract presented at a scientific conference – something that has not yet undergone the kind of peer review that a journal would use.
This story relies too heavily on researchers who are connected to the companies developing the cancer screens. Where was the truly independent perspective?
As noted in the “benefits” criterion above, this story did not even discuss other stool tests like fecal immunochemical testing or other fecal occult blood tests that are currently available and have been shown to perform well in appropriate samples. It did a decent job in comparing two new technologies, but got hung up on this comparison while forgetting existing tests. In so doing, the end of the story devolves a bit into a “my product is better than your product” tit for tat. Which is what can happen when your only quoted sources all have connections to the companies involved.
Although the story doesn’t discuss existing stool tests like fecal immunochemical testing or other fecal occult blood testing, on the issue of relative novelty, the story scores points for being the only we reviewed that mentioned that the Exact Sciences DNA test is only one of two leading technologies being developed right now.
This story did not rely on a news release.
Systematic, Criteria-Driven
The tale of two “negative” studies. Or, how spin is applied to make research findings look more impressive. A must read. https://johnmandrola.substack.com/p/the-tale-of-two-negative-studies
The @nytimes chose to interview one of the editors of the @AnnalsofIM (publisher of this paper) who "gushed with extraordinary glee: It’s huge! There are very few things that reduce your mortality by 30%."
Turns out coffee is not one of those few things despite all that glee. https://twitter.com/garyschwitzer/status/1533202075928215558
Comments
Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.
Our Comments Policy
But before leaving a comment, please review these notes about our policy.
You are responsible for any comments you leave on this site.
This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.
We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.
We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.
There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”
The authors of the Retraction Watch comments policy urge commenters:
We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.
And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.
The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.
You might also like