Michael Joyce is a writer-producer with HealthNewsReview.org and tweets as @mlmjoyce
It’s hard to believe that the above functional magnetic resonance imaging (fMRI) image could lead to headlines like this:
Can suicide be prevented by science? (International Business Times)
It’s also hard to believe — or maybe it’s not — that this study generated all this hype. Here’s what the researchers from Carnegie Mellon University and University of Pittsburgh did to spark these headlines:
What’s lost in the breathless coverage generated by this study are two very important limitations.
First, none of the coverage I reviewed brought up the surrogate nature of fMRI. That what a fMRI measures is simply blood flow. And that the activity of neurons in the brain is correlated with blood flow. That is, blood flow increases to parts of the brain we are actively using. What lights up does not equate with a complex behavior or mood.
Just because researchers present patients with words or concepts they themselves associate with suicide (or whatever the opposite of suicide is?), does not mean they can draw any conclusions regarding what the subjects are thinking or feeling. Specifically, the findings can not predict suicidality as some headlines implied.
“There is an enormous gap between pixels on a scan and the suicidal behavior we see clinically,” says Allen Frances MD, former chairman of the psychiatry department at Duke University, who thought publishing the study was questionable. “Also, preliminary fMRI findings almost always fail to replicate when larger samples and more realistic control groups are used.”
Frances’s comments hint at the second key limitation of this study, which is the extremely small sample size. Although most of the news coverage mentioned the tiny sample size, none mentioned the dropout rate of nearly 50 percent, and the fact that most of the subjects were young females. It will take a much larger study to know if the results can be generalized more broadly.
That a computer can recognize a pattern — and these patterns can be sorted to generate an algorithm — is not surprising. What is surprising is what study co-leader, Marcel Just PhD, said in this video embedded in the Carnegie Mellon University news release :
I find three statements Dr. Just makes in this video disconcerting (italicized/underlined for emphasis):
I think the words he uses create problems for both journalists and readers. I also think they could foster hype.
Word choices do matter. We’ve run into this before with communicating about fMRI studies.
Just’s comments suggest that a brain imaging technique like fMRI can actually see and measure thoughts (or emotions). And to suggest this, as well as claim the study results are ‘revolutionary’ — or might help guide therapy — is misleading, irresponsible, and potentially self-serving. The gap between these scans and any possible therapy for mental illness is enormous.
And reporters have a responsibility here too. The International Business Times opened with: “Suicidal individuals can be identified through a new algorithm … a potential method for diagnosing mental health conditions in the near future.” What are suicidal people — who are likely quite desperate — expected to do with this information? It’s irresponsible reporting.
I empathize with fMRI researchers trying to find a vocabulary for what they see. Words like “signatures” or “patterns” make sense to me. But once you start to slide over to speaking about the brain’s ‘representation’ of words, thoughts, and emotions I think you run the risk of misleading people. You imply causality when you know full well as a scientist that your results can suggest correlation at best. In other words, we can’t know if the areas that light up in suicidal people represent their depression or intention to hurt themselves.
“I’m sure this is interesting research to those in the field of imaging and cognition,” says psychiatrist and former brain imaging researcher, Susan Molchan MD, a frequent contributor on our site. “But as far as clinical usefulness it has none, and likely won’t for a very long time.”
But a study like this does serve as a useful reminder to journalists and consumers alike. It reminds us that, other than the necessity of scrutinizing study limitations and word choices, it has becoming increasingly necessary to scrutinize news releases. It doesn’t matter if they come from corporations, medical journals, or — as in this case — from respected and trusted academic institutions.
All too often the burden of proof is left on the shoulders of journalists (and readers) to ask: what master does this news release serve? Is it leaning toward self-promotion or promoting evidence-based information?
Because I found the statistical analysis of this study somewhat confusing, I reached out to one of our frequent contributors, Susan Wie PhD, of the Division of Biostatistics at the University of Minnesota School of Public Health. She sent me the following which I think provides another reason to approach this study with further skepticism:
The reported high accuracy of 0.91 in the abstract was assessed in a non-standard way. The gold standard, which uses something called cross-validation, reveals a much lower classification accuracy of 0.76. In other words, the generalizability of this algorithm for identifying suicidality should be taken with a grain of salt.”