Perhaps the most troubling example yet of early reporting of abstracts released far in advance of the American Society of Clinical Oncology meeting. Read this review to see why this is a big issue.
As the ratings show, this story about a new genetic test for colon cancer falls short of journalistic best practices in several ways. Among other things it fails to report costs, explain the results clearly, provide necessary caveats or put the test in context of clinical use.
Worse than that, viewed as a whole the story is simply misleading.
The current study demonstrates that the test, Genomic Health’s Oncotype DK, can help identify patients at lower and higher risks of recurrence. But the study failed to show that the test could identify people more likely to benefit from chemotherapy.
Several other news reports treated these results with care: Sources in those stories are quoted saying that since the test can identify only higher and lower risk of recurrence, it might be useful as "a tie-breaker" in a yes-or-no-chemo decision; another says that it’s "not as powerful as we’d like" in order to be of significant clinical value; a third that "it’s a first step"; yet another calls it "half a loaf."
Several stories included in the lead or high up in the story the important fact that the test was not proven to identify people more likely to benefit from treatment.
Why did this story take such a different approach, allowing sources to imply the test has clinical value in the absence of evidence?
Possibly because the story didn’t include quotes from any truly independent sources. Given the fact that the study is unpublished and financed by the test maker, this is an alarming omission.
And what are the consequences of this sort of incomplete reporting?
After this story appeared in the printed Wall Street Journal in the morning, shortly after 3 p.m. the same afternoon the publication’s Health Blog reported the following:
Genomic Health was trading higher on data suggesting its test for early-stage colon cancer may help patients and doctors decide whether they needed chemotherapy after their tumors are removed with surgery. Read the WSJ story on the colon-cancer test here.
Supposedly, in some way, the early release of abstracts from the American Society of Clinical Oncoloy meeting, according to another story by the WSJ, was supposed to avoid stock manipulation.
We are troubled by the early release – which heightens speculation – and by the reporting on that speculation.
A company official declines to estimate how much the test will cost. This information is in the last line of the story.
It’s tempting to assign a satisfactory rating for at least asking the question.
But a more complete story would have asked the official to estimate a price compared to the company’s similar breast cancer test now on the market.
In fact, one did: A Bloomberg story reported that the company’s breast cancer test costs about $3,800.
An even more complete story, from the Associated Press, mentioned that despite their price insurers often pay for some tests like this because they may be able to avoid spending $30,000 to $40,000 for chemotherapy on patients who do not benefit.
Reporting costs is important for at least two reasons:
The story makes an astonishing mess of the key data:
After more than six years of tracking, the study found that 44% of the patients had tumors that had a 12% risk of recurrence within three years, which the researchers considered low risk. An additional 26% had a three-year recurrence risk of 22%, which researchers put in the high-risk group. The rest were in the middle.
Even a reader lucky enough to have within easy reach a calculator and a high school student taking advanced statistics is unlikely to figure out what this means in practical terms.
But this is what you get when you report on abstracts far in advance of any possible publication.
Because the study failed to prove that the Oncotype DX test could identify the patients most likely to benefit from chemotherapy, the risk is that the test would be used for this purpose anyway.
This triggers the risk of "false reassurance"–that a resassuring test result would lead to no aggressive treatment yet a bad outcome.
The story fails to warn about the potential risks of clinical over-use and over-reliance on a test that may not improve survival rates.
The story states that the company has "unveiled" results of the study and they will be presented at an upcoming oncology conference. This is another story based on an incomplete abstract released far in advance of the American Society of Clinical Oncology meeting that hasn’t even taken place yet.
The story fails to include the caveats that the study has not been peer-reviewed and has not been published.
In a study funded by the test’s maker, it’s essential to mention those facts.
Further, the study–a retrospective review of patients whose tumor biology was analyzed and a predictive model created from those analyses–lacks the power of a prospective clinical trial. The story should have said this.
The reporter does a good job describing the prevalence, incidence and mortality of the disease without exaggeration.
The story quotes three sources:
Given the fact that [as the story discloses] the study was paid for by the test maker, the lack of independent sources is glaring.
The story would have been much stronger if it had comments from a disinterested oncologist who could speak to the test’s practical value in a clinical setting.
The story explains that the current alternative option to the genetic test is to assess risk of recurrence based on a patient’s age, other illnesses and tumor characteristics–a method that results in a clear recommendation for or against chemo in only 30 percent of patients.
The story says the company that makes the Oncotype DX test, Genomic Health, plans to start selling the test early next year.
But the reader has no idea how realistic this prediction is. Given the economic self-interest of the source, the story should have indicated what regulatory or commercial barriers the company faces before it can bring the test to market.
The story makes clear that the test is similar to the company’s genetic test to identify more lethal breast cancers, yet that it is the first such test for colon cancer.
The story does not appear to draw excessively from the company’s press release.