Promise Seen in Drug for Retardation Syndrome

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

Again, the story falls down here when it passes along a Novartis researcher’s vague account of what happened in the study instead of providing us with actual statistics.  The researcher says there was "clear improvement" in the treated patients, but the story should have pressed for a more detailed account of how many patients benefited, by how much, and according to what outcome.
 
Also unsatisfactory was the rampant speculation about who else might benefit from the drug beyond patients with fragile X syndrome. According to the Novartis account of the research, only a subgroup of patients with an "undisclosed biological trait" benefited from therapy with the experimental drug. And this was in a study that was already extremely small ("a few dozen patients") to begin with. But The Times quotes three different sources who suggest, based on no evaluation of the data, that the drug might be useful for a broader spectrum of autism disorders. And the quotes include some over-the-top language about the drug perhaps being "the key to solving the mystery of autism and other developmental disorders."

By relentlessly playing up the positives and failing to rein in these more exuberant proponents, we think the story gives an unbalanced view of progress in the development of this drug and what it will likely be capable of. The fact is that very few drugs in this stage of development will be approved by the FDA and become available to patients.   

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

One of the big problems of reporting on research that hasn’t been published or peer-reviewed is that harms don’t get adequately quantified–a  drawback that was on full display in this story. The report didn’t even mention the possibility of patients being harmed by this experimental drug.  

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

The story does an excellent job of explaining the science behind fragile X syndrome and the discovery process that led to the new investigational drug. And at first glance, the story seems to call attention to all the appropriate caveats and limitations that apply to early-stage drug research. We are told that the drug is years away from approval and that it might fail in future clinical trials. The Times also warns us that the trial results were not published or peer reviewed. 
 
Troubling, however, is the fact that other researchers who commented for the story were doing so without having reviewed the Novartis data and had to be told by the reporter what the study found. That doesn’t strike us as a recipe for collecting useful feedback about complex medical research. Also problematic is the story’s characterization of the study outcomes reported by the researchers. We are told that an "undisclosed biological trait" improved in the treated group more than in the control group. The story says that these and other details about the study could not be disclosed by Novartis for commercial reasons. So the question inevitably arises: If the limitations and restrictions on a study are such that a reporter can’t get independent feedback about it and can’t even tell us what benefits were found, is the newspaper really justified in spending 1800 words covering the story? If the only evidence you are allowed to report on is a company researcher’s assurance that "clear improvements in behavior" were found, we think it’s a good sign the story might need a little more seasoning before it gets splashed across the front page.

Does the story use independent sources and identify conflicts of interest?

Not Satisfactory

While the story does collect feedback from a number of independent expert sources, we think the way these sources were used was inappropriate and potentially misleading. The reporter seems to have approached many of the sources and asked them to comment– without the benefit of any actual data–on the reporter’s second-hand description of the apparent positive results of Novartis’s research. The sources all respond with a variation on, "Wow, that sounds wonderful." And really, who wouldn’t say the same thing? Nobody doubts that a drug effective for fragile X syndrome would represent a huge medical breakthrough. But the more important question prompted by the article is whether such a breakthrough has in fact occurred. Although there are some cautions and caveats sprinkled throughout the article, we think the extremely positive tone of the expert quotes far outweighs these nuggets of restraint. What bears repeating (and what we think gets lost in this story) is that these experts are enthusiastic about the concept of an effective treatment; they are not endorsing Novartis’s claim that the drug caused "clear improvements" in behavior.