A story that was flawed at its core – treating the outcome of not-yet-done studies in children as a fait accompli. The three-star score is deceptively high.
The enthusiasm this story effuses about a new brain scanning technology to diagnose autism may well be warranted. But it fails to give more than the most cursory of cautions about the early stage of this research and the tough road that lies ahead. The story makes it sound like we are a slam dunk away from seeing this test at the local radiology clinic. In reality, much larger studies with a much greater range of patients will be needed to tell whether the scan is effective enough to be widely used.
Autism is difficult to diagnose due to its many different causes and symptoms and because it often occurs with other similar neurodevelopmental disorders. A fast and accurate diagnostic brain scan would take much of the uncertainty out of the diagnostic process and would hopefully speed treatment and other support services to patients.
Many of the deficiencies discussed above in the "Evidence" criterion could also be applied to this criterion. Importantly, the story didn’t do enough to call attention to the small and highly selected sample of patients, which likely biases the results towards greater accuracy than what would be observed in a larger and more diverse sample. There was also inadequate acknowledgment of the fact that this was a study of adults whereas the real need is for diagnostic tool for use in children. Yes, it was acknowledged that the childrens’ studies hadn’t been done but the outcome of these not-yet-done studies was almost treated as a fait accompli. Lastly, we would like to have seen some discussion of a somewhat subtle but important issue that wasn’t addressed. That is, whenever we diagnose a condition with new technology, the group of individuals identified as having the condition, in this case autism, will change. Some previously undiagnosed children will be labeled with autism and some previously labeled with autism will be told that this is not the diagnosis. In addition, the severity of the disease may change (a more sensitive test will likely identify milder cases). Anytime the spectrum of disease changes, we have to ask ourselves, Will the treatments produce the same benefits and harms? Since benefits are often smaller for individuals with milder disease but the adverse effects are the same, the risk benefit ratio often increases with the more sensitive test.
As discussed above, the story doesn’t attempt to quantify the harm of false-positive or false-negative diagnoses. It also doesn’t explore the potential harm of labeling someone with borderline symptoms as having autism. In addition, it has now become clear that MRI with contrast is associated with kidney damage. Physicians now check kidney function in all patients prior to an MRI and must weigh the need for the images vs the very small (but tangible) risk of kidney damage.
Readers would have been better served by more detail and less unbridled optimism in this piece. To wit: we learn that the scan’s accuracy was "so high" that the results were strongly significant despite the study’s small sample size. Instead of gushing over the test’s accuracy, however, we think the story should have spent more time discussing the limitations of such a small study and the additional research that needs to be conducted to confirm these results. We still don’t know whether the method can differentiate between autism and other neurodevelopmental conditions. And it is unclear whether the results are applicable beyond the small group of high-functioning adult males who were studied.
We also think the story failed to provide all of the necessary information about the study’s outcomes. To be effective, diagnostic tests must do more than simply identify individuals with a disease (which is the result this story focused on); they must also rule out those individuals who do not have a disease and do not require treatment. Failure to accomplish the latter will result in an excessively high false-positive rate and could result in much unnecessary treatment of healthy people. We recognize that this is a complex concept to convey in a consumer news story, but the story didn’t even mention the possibility that people without autism could be given an erroneous positive diagnosis.
Finally, we should call attention to the story’s speculation about applying the new technology to children. The story quotes one of the study investigators saying there is "no reason" why the test wouldn’t work in children, even though it has never been tested in this population. In fact, there are any number of good reasons why this test might be less effective in the developing brain than in a mature one. Someone involved in this story – the reporter? the editor? somebody, please! – should have known this and injected some skepticism into the discussion.
The story didn’t exaggerate the prevalence or impact of autism.
One of the sources for this story was not involved in the research and has no obvious conflicts of interest.
The story notes that autism is currently diagnosed via interviews and patient observation.
Although it is clear that this technology is not yet available to the public, the story predicts the new scan "could be ready for general use in a couple of years." Really? Who says? It seems unlikely that any knowledgeable researcher would make such a wildly optimistic prediction on the basis of a 40-person study. But the story treats this forecast as an established fact requiring no attribution, so we can’t tell where it came from. In any case, we think such pie-in-the-sky guesstimates are usually wrong and should be avoided in health journalism.
Researchers have long studied the brains of patients with autism using MRI scans. However, the ability to make a clinical diagnosis of autism using MRI would be noteworthy progress.
Since this story included two expert interviews, we can be sure it didn’t rely on a news release