Quick brain scan could screen for autism

Does the story adequately discuss the costs of the intervention?

Not Satisfactory

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

Many of the deficiencies discussed above in the "Evidence" criterion could also be applied to this criterion. Importantly, the story didn’t do enough to call attention to the small and highly selected sample of patients, which likely biases the results towards greater accuracy than what would be observed in a larger and more diverse sample. There was also inadequate acknowledgment of the fact that this was a study of adults whereas the real need is for diagnostic tool for use in children. Yes, it was acknowledged that the childrens’ studies hadn’t been done but the outcome of these not-yet-done studies was almost treated as a fait accompli. Lastly, we would like to have seen some discussion of a somewhat subtle but important issue that wasn’t addressed. That is, whenever we diagnose a condition with new technology, the group of individuals identified as having the condition, in this case autism, will change. Some previously undiagnosed children will be labeled with autism and some previously labeled with autism will be told that this is not the diagnosis. In addition, the severity of the disease may change (a more sensitive test will likely identify milder cases). Anytime the spectrum of disease changes, we have to ask ourselves, Will the treatments produce the same benefits and harms? Since benefits are often smaller for individuals with milder disease but the adverse effects are the same, the risk benefit ratio often increases with the more sensitive test.    

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

As discussed above, the story doesn’t attempt to quantify the harm of false-positive or false-negative diagnoses. It also doesn’t explore the potential harm of labeling someone with borderline symptoms as having autism. In addition, it has now become clear that MRI with contrast is associated with kidney damage. Physicians now check kidney function in all patients prior to an MRI and must weigh the need for the images vs the very small (but tangible) risk of kidney damage.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

Readers would have been better served by more detail and less unbridled optimism in this piece. To wit: we learn that the scan’s accuracy was "so high" that the results were strongly significant despite the study’s small sample size. Instead of  gushing over the test’s accuracy, however, we think the story should have spent more time discussing the limitations of such a small study and the additional research that needs to be conducted to confirm these results. We still don’t know whether the method can differentiate between autism and other neurodevelopmental conditions. And it is unclear whether the results are applicable beyond the small group of high-functioning adult males who were studied.  

We also think the story failed to provide all of the necessary information about the study’s outcomes. To be effective, diagnostic tests must do more than simply identify individuals with a disease (which is the result this story focused on); they must also rule out those individuals who do not have a disease and do not require treatment. Failure to accomplish the latter will result in an excessively high false-positive rate and could result in much unnecessary treatment of healthy people. We recognize that this is a complex concept to convey in a consumer news story, but the story didn’t even mention the possibility that people without autism could be given an erroneous positive diagnosis. 

Finally, we should call attention to the story’s speculation about applying the new technology to children. The story quotes one of the study investigators saying there is "no reason" why the test wouldn’t work in children, even though it has never been tested in this population. In fact, there are any number of good reasons why this test might be less effective in the developing brain than in a mature one. Someone involved in this story – the reporter?  the editor?  somebody, please! – should have known this and injected some skepticism into the discussion.

Does the story establish the availability of the treatment/test/product/procedure?

Not Satisfactory

Although it is clear that this technology is not yet available to the public, the story predicts the new scan "could be ready for general use in a couple of years." Really? Who says? It seems unlikely that any knowledgeable researcher would make such a wildly optimistic prediction on the basis of a 40-person study. But the story treats this forecast as an established fact requiring no attribution, so we can’t tell where it came from.  In any case, we think such pie-in-the-sky guesstimates are usually wrong and should be avoided in health journalism.