Mary Chris Jaklevic is a freelance health reporter who joined our review team in April 2016. She tweets as @mcjaklevic
We often call out overly optimistic news coverage of drugs and devices. But information technology is another healthcare arena where uncritical media narratives can cause harm by raising false hopes and allowing costly and unproven investments to proceed without scrutiny.
A case in point is the recent collapse of M.D. Anderson Cancer Center’s ambitious venture to use IBM’s Watson cognitive computing system to expedite clinical decision-making around the globe and match patients to clinical trials.
Launched in 2013, the project initially received glowing mainstream media coverage that suggested Watson was already being deployed to revolutionize cancer care–or soon would be.
But that was premature. By all accounts, the electronic brain was never used to treat patients at M.D. Anderson. A University of Texas audit reported the product doesn’t work with Anderson’s new electronic medical records system, and the cancer center is now seeking bids to find a new contractor.
The audit chronicles rife financial missteps by the cancer center including no competitive bidding, payments to vendors without documented results, and decision-making that skirted the medical center’s own IT department. The bungle cost $62 million paid to IBM and PricewaterhouseCoopers, plus uncalculated internal resources including staff time, technology infrastructure and administrative support, according to the audit. The cost was supposed to be shouldered by a single donor who didn’t come through with all of the promised funds.
It’s uncertain how much of the failure was the fault of M.D. Anderson versus the limitations of the technology. But looking at early news coverage, it is clear journalists were not asking pointed questions about how the project was being financed or its capabilities when it comes to curing cancer.
The dominant narrative was that the technology that famously bested two human champions on the game show Jeopardy! in 2011 was being re-deployed to augment cancer care. Only limited play was given to the complexity of integrating medical research and patient records to craft an effective decision-making tool.
A 2015 Washington Post story entitled “Watson’s next feat? Taking on cancer. IBM’s computer brain is training alongside doctors to do what they can’t,” mentioned some limitations of machine learning but took an overall positive tone. It describes Watson as “a revolutionary approach to medicine and health care that is likely to have significant social, economic and political consequences.”
The story said also Watson would enable doctors “to find personalized treatments for every cancer patient by comparing disease and treatment histories, genetic data, scans and symptoms against the vast universe of medical knowledge.”
But treating cancer is more complex than winning a trivia game, and the “vast universe of medical knowledge” may not be as significant as purveyors of artificial intelligence make it out to be, according to some observers.
“IBM spun a story about how Watson could improve cancer treatment that was superficially plausible – there are thousands of research papers published every year and no doctor can read them all,” said David Howard, a faculty member in the Department of Health Policy and Management at Emory University, via email. “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.”
While Watson’s use in cancer care was still developing, a 2013 IBM news release declared MD Anderson “is using the IBM Watson cognitive computing system for its mission to eradicate cancer.”
From what we know now, the system was a long way from being used for patient care. Yet some media reports echoed this premature language, suggesting it was or soon would be operational.
Forbes ran a blog headlined “IBM’s Watson Now Tackles Clinical Trials At MD Anderson Cancer Center.” Forbes stated use in patient care “might come in early 2014.” It quoted an M.D. Anderson doctor saying: “It’s still in testing and not quite ready for the mainstream yet, but it has the infrastructure to potentially revolutionize oncology research.”
Likewise Scientific American asserted: “The University of Texas M.D. Anderson Cancer Center is using Watson to help doctors match patients with clinical trials, observe and fine-tune treatment plans, and assess risks as part of M. D. Anderson’s ‘Moon Shots’ mission to eliminate cancer.”
The magazine gave the project another PR boost by publishing a blog post by IBM Chief Technology Officer Rob High and financier Jho Low, whose foundation funded the project, entitled “Expert Cancer Care May Soon Be Everywhere, Thanks to Watson.”
Largely missing in this coverage was a caveat about the lack of evidence that the technology improved patient outcomes, lowered costs, or provided some other benefit — something we demand in reporting on drugs, devices and tests.
“Reporters are often susceptible to PR hype about the potential of new technology – from Watson to ‘wearables’ – to improve outcomes,” Howard said. “A lot of stories would turn out differently if they asked a simple question: ‘Where is the evidence?’”
At this point, there isn’t much. While IBM has entered into numerous deals to use its artificial intelligence system in healthcare, a company spokeswoman said there’s no published study linking the technology to improved outcomes for patients because “the implementation of the technology is not there yet.”
She did cite published studies showing the system met operational objectives, such as matching clinicians’ treatment recommendations in a given percentage of cases.
When it comes to IT coverage, journalists should make a habit of pointing out gaps between what’s claimed and what’s been demonstrated to work.
“Artificial intelligence has been suffering from overhype since the 1970s and 80s,” said Steven Salzberg, a professor of biomedical engineering at the Johns Hopkins School of Medicine. “Be skeptical and ask to see some evidence. (Technology companies) need to do more than simply assert that it works.”