Strengths: Immediately, in the opening words, the story acknowledges that there is another DNA test in development that was reported on this week. It was the only one of the four stories we reviewed on this topic that made this clear.
Weaknesses: It didn’t provide a meaningful comparison of the new approaches with existing colon cancer screening tests.
It also has problems with how the underlying question is presented. Two examples:
This topic is really about when we should pay attention to evidence about a new diagnostic test and how we should evaluate diagnostic tests in comparison with one another- a topic that is often inadequately discussed in both medicine and journalism.
The story says, “Both tests would be less expensive than colonoscopy, and potentially more effective.” No specific cost estimate is given. The Philadelphia Inquirer said “Though the cost has not been set, it could be as low as $300 per test.” Reuters reported that the company said earlier that it would aim for $300-400 per test.
The story also states that “Compliance with colonoscopy is low, since people don’t want to have one, and the overall cost per detection is high” – implying that the new test might be more cost-effective. But it’s unclear at this point if that would be true. For example, previous stool DNA tests have performed much worse (higher costs, lower efficacy) than colonoscopy in modeling studies.
The story did a decent job comparing data presented on two new competing tests. It does not, however, compare the benefits to existing colon cancer screening tests. We will dock points for this in our final criterion on “comparing the new approach with existing alternatives” but will give it the benefit of the doubt on this one.
The two tests the Times reported on were done on blood and on stool, which, in theory, should not harm anyone during the screening. But there is potential for harms as a result of screening. The story states that with the Exact Sciences DNA test “12 percent of the time the patient will be given a false alarm.” And, regarding the Epigenomics blood test, the story states that “more false positives could be generated so you could end up doing more harm than good.”
Mixed bag. The story did provide some important context about test. For example, it says, “Exact Sciences reported in July that its test was highly sensitive and specific when applied directly to cells taken from tumors. But in the real world, the tumor DNA must be detected in stool samples, even though almost all the DNA comes from the bacteria of the gut. Just 0.01 percent of the DNA is human, and most of this is normal DNA, not the altered DNA of tumors.”
But only in passing does it mention that “in the real world, the tumor DNA must be detected in stool samples.” It could/should have driven home the point more clearly that the methodology used in the study tends to overestimate performance compared with an evaluation done in a representative sample population (hence the need to do another study). WebMD, for example, did a better job on this point, with a quote from Dr. Durado Brooks of the American Cancer Society: “This was a very preliminary study that utilized samples from individuals who were known to have colon polyps or cancer. The performance may decrease considerably when the test is used in a large population of healthy individuals.” Health Day quoted Brooks: “Showing that in a small group of samples is very different from demonstrating that in a population where only a small number of individuals are going to have polyps of that size. Then we will know if this is a big step forward.”
There was also no discussion of the limitations of drawing conclusions from an abstract presented at a scientific conference – something that has not yet undergone the kind of peer review that a journal would use.
No disease mongering.
This story relies too heavily on researchers who are connected to the companies developing the cancer screens. Where was the truly independent perspective?
As noted in the “benefits” criterion above, this story did not even discuss other stool tests like fecal immunochemical testing or other fecal occult blood tests that are currently available and have been shown to perform well in appropriate samples. It did a decent job in comparing two new technologies, but got hung up on this comparison while forgetting existing tests. In so doing, the end of the story devolves a bit into a “my product is better than your product” tit for tat. Which is what can happen when your only quoted sources all have connections to the companies involved.
Although the story doesn’t discuss existing stool tests like fecal immunochemical testing or other fecal occult blood testing, on the issue of relative novelty, the story scores points for being the only we reviewed that mentioned that the Exact Sciences DNA test is only one of two leading technologies being developed right now.
This story did not rely on a news release.