MD Anderson Cancer Center’s IBM Watson project fails, and so did the journalism related to it

Mary Chris Jaklevic is a freelance health reporter who joined our review team in April 2016. She tweets as @mcjaklevic

We often call out overly optimistic news coverage of drugs and devices. But information technology is another healthcare arena where uncritical media narratives can cause harm by raising false hopes and allowing costly and unproven investments to proceed without scrutiny.

A case in point is the recent collapse of M.D. Anderson Cancer Center’s ambitious venture to use IBM’s Watson cognitive computing system to expedite clinical decision-making around the globe and match patients to clinical trials.

Launched in 2013, the project initially received glowing mainstream media coverage that suggested Watson was already being deployed to revolutionize cancer care–or soon would be. 

But that was premature. By all accounts, the electronic brain was never used to treat patients at M.D. Anderson. A University of Texas audit reported the product doesn’t work with Anderson’s new electronic medical records system, and the cancer center is now seeking bids to find a new contractor.

The audit chronicles rife financial missteps by the cancer center including no competitive bidding, payments to vendors without documented results, and decision-making that skirted the medical center’s own IT department. The bungle cost $62 million paid to IBM and PricewaterhouseCoopers, plus uncalculated internal resources including staff time, technology infrastructure and administrative support, according to the audit. The cost was supposed to be shouldered by a single donor who didn’t come through with all of the promised funds.

It’s uncertain how much of the failure was the fault of M.D. Anderson versus the limitations of the technology. But looking at early news coverage, it is clear journalists were not asking pointed questions about how the project was being financed or its capabilities when it comes to curing cancer.

Overwhelmingly positive coverage of Watson ‘taking on cancer’

The dominant narrative was that the technology that famously bested two human champions on the game show Jeopardy! in 2011 was being re-deployed to augment cancer care. Only limited play was given to the complexity of integrating medical research and patient records to craft an effective decision-making tool.

A 2015 Washington Post story entitled “Watson’s next feat? Taking on cancer. IBM’s computer brain is training alongside doctors to do what they can’t,” mentioned some limitations of machine learning but took an overall positive tone. It describes Watson as “a revolutionary approach to medicine and health care that is likely to have significant social, economic and political consequences.”

The story said also Watson would enable doctors “to find personalized treatments for every cancer patient by comparing disease and treatment histories, genetic data, scans and symptoms against the vast universe of medical knowledge.”

But treating cancer is more complex than winning a trivia game, and the “vast universe of medical knowledge” may not be as significant as purveyors of artificial intelligence make it out to be, according to some observers.

The problem isn’t too much data for humans–it’s too little

“IBM spun a story about how Watson could improve cancer treatment that was superficially plausible – there are thousands of research papers published every year and no doctor can read them all,” said David Howard, a faculty member in the Department of Health Policy and Management at Emory University, via email. “However, the problem is not that there is too much information, but rather there is too little. Only a handful of published articles are high-quality, randomized trials. In many cases, oncologists have to choose between drugs that have never been directly compared in a randomized trial.”

While Watson’s use in cancer care was still developing, a 2013 IBM news release declared MD Anderson “is using the IBM Watson cognitive computing system for its mission to eradicate cancer.”

From what we know now, the system was a long way from being used for patient care. Yet some media reports echoed this premature language, suggesting it was or soon would be operational.

Forbes ran a blog headlined “IBM’s Watson Now Tackles Clinical Trials At MD Anderson Cancer Center.” Forbes stated use in patient care “might come in early 2014.” It quoted an M.D. Anderson doctor saying: “It’s still in testing and not quite ready for the mainstream yet, but it has the infrastructure to potentially revolutionize oncology research.”

Likewise Scientific American asserted: “The University of Texas M.D. Anderson Cancer Center is using Watson to help doctors match patients with clinical trials, observe and fine-tune treatment plans, and assess risks as part of M. D. Anderson’s ‘Moon Shots’ mission to eliminate cancer.”

The magazine gave the project another PR boost by publishing a blog post by IBM Chief Technology Officer Rob High and financier Jho Low, whose foundation funded the project, entitled “Expert Cancer Care May Soon Be Everywhere, Thanks to Watson.”

No one seemed to ask does it improve patient outcomes? Lower costs? 

Largely missing in this coverage was a caveat about the lack of evidence that the technology improved patient outcomes, lowered costs, or provided some other benefit — something we demand in reporting on drugs, devices and tests.

“Reporters are often susceptible to PR hype about the potential of new technology – from Watson to ‘wearables’ – to improve outcomes,” Howard said. “A lot of stories would turn out differently if they asked a simple question: ‘Where is the evidence?’”

At this point, there isn’t much. While IBM has entered into numerous deals to use its artificial intelligence system in healthcare, a company spokeswoman said there’s no published study linking the technology to improved outcomes for patients because “the implementation of the technology is not there yet.”

She did cite published studies showing the system met operational objectives, such as matching clinicians’ treatment recommendations in a given percentage of cases.

When it comes to IT coverage, journalists should make a habit of pointing out gaps between what’s claimed and what’s been demonstrated to work.

Artificial intelligence has been suffering from overhype since the 1970s and 80s,” said Steven Salzberg, a professor of biomedical engineering at the Johns Hopkins School of Medicine. “Be skeptical and ask to see some evidence. (Technology companies) need to do more than simply assert that it works.”

You might also like

Comments (7)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Tom Mariner

February 23, 2017 at 5:00 pm

Yes, everyone took credit before the product was “shipped”. Yes, the rules of procurement were bent, but there was no choice about IBM and none about PwC who got virtually all of the funds — they are one company. Yes, no one coordinated with the IT department that owned the patient records.

But the biggest failure of all was not doing a Medical Device Commercialization. It would have taken far less than the five years, and would have WORKED. If some useful results would have been obtained for the $70 Million, we wouldn’t be having this conversation and throwing stones at everyone involved.

Carlos A Selmonosky,MD

February 24, 2017 at 4:23 pm

A statement that every physician should know”In many cases the oncologists( I assume most of the physicians) have to choose between drugs that have never directly compared( to other drugs manufactured by the same company ) in a randomized trial).
Most of the new drugs are compared to “controls” or ineffective theraphies,Also inneffectual drug trials are not widely reported.

Mark Montgomery

February 24, 2017 at 7:07 pm

While I certainly agree with and appreciate the commentary on media reporting and auto reporting on Watson, this is also a good example of why many of us who have been through these cycles and work in the trenches warn about over-hype.

This sort of story becomes a poster and excuse for those attempting to block more efficient systems and, in this case, it could harm an enormous number of people. It takes some real research and time, but there are many good public works available now in highly specific use cases under the umbrella (admittedly not terribly accurate label) of AI.

For a sense of my frustration, take a look at this post from over a year ago on an article in the WSJ citing this project:

Why go to the Moon when what your company really needs is in the Rockies? (AI, Watson, Kyield)
https://kyield.wordpress.com/2015/12/06/why-go-to-the-moon-when-what-your-company-really-needs-is-in-the-rockies-ai-watson-kyield/

There is an awful lot of good work out there not getting much if any light–and by extension the chance to demonstrate efficacy at a fraction of these costs, apparently simply due to a combination of marketing spend, click bait and keyword ad models, and media herding. An opportunity exists for those willing to do some work.

    Gary Schwitzer

    February 24, 2017 at 7:33 pm

    And our site, as is clear throughout our nearly 11-year history, primarily addresses media reporting.

    Gary Schwitzer
    Publisher

bev M.D.

February 27, 2017 at 4:06 am

Reading this article in the context of our national politics makes me wonder why journalists repeatedly ‘fall for’ the medical hype, and yet are so skeptical when it comes to politics. One would think they could carry this skeptical attitude over to an industry which, increasingly deserves skepticism.

Fred Roycroft

March 6, 2017 at 8:19 am

Watson’s win on Jeopardy wasn’t as straightforward as everyone thinks. Contrary to public perception, Watson has never had a speech interface. So for Jeopardy the questions were submitted in written form to Watson. However, the way the game was played, Watson received the question as soon as Alex Trebek began reading the question to the other contestants. With the speed that computers process information this meant that Watson had something like an hour to contemplate the question before the other contestants had finished hearing the last words. With this type of advantage it’s no surprise that Watson won. And IBM’s marketing department has taken that golden ring and run with it ever since.