Read Original Release

Sweeping claims based on a single case study of advanced cancer patient are misleading

NUS researchers use AI to successfully treat metastatic cancer patient

Our Review Summary

artificial intelligenceThis news release focuses on a paper from Singapore-based researchers who used artificial intelligence (AI) to treat cancer in a single patient. The release argues that this is a “gamechanger” and a “big step forward” that “is applicable to all diseases and all patients” — which are bold claims to make based on the treatment of only one patient.

Unjustifiable language throughout and a lack of attention to key details like harms, cost and an explanation of the severe limitations of the research make this a weak and misleading release.

 

Why This Matters

There has been a lot of research focused on the use of AI to help diagnose cancers and develop treatment protocols — and it continues to be an area of great interest. However, it is difficult — if not impossible — to draw far-reaching conclusions from a single case. What costs, benefits and risks are associated with AI-directed treatment for cancers? A study like the one highlighted in this release simply cannot tell us.

The release claims the researchers were able to improve treatment efficacy by adjusting treatment dose and presumably still minimizing adverse events. Again — impossible to prove based on a single patient.

It is an interesting case study, and we look forward to learning the results of future research, but it is misleading (if not dishonest) to make sweeping claims based on the treatment of a single patient.

 

Criteria

Does the news release adequately discuss the costs of the intervention?

Not Satisfactory

The AI used in this case is called the “CURATE.AI” platform. It’s not clear if the platform is proprietary. If it is, it is not clear what the cost would be for health care providers to access and make use of the platform. It is also not clear what level of training, and associated costs, would be needed for health care providers to be able to use the platform in a clinical setting. Given that the AI platform appears to be designed to create personalized treatments for each patient, the cost could be considerable. It’s hard to estimate, since the release does not address cost in any way.

Does the news release adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The release does not attempt to quantify benefits. In fact, it would be impossible for the release to address this issue in any meaningful way, given that the relevant study was of one patient. The release says that the patient was “successfully treated” — but what does that mean? Lower down in the release, readers are told that “disease progression was halted,” meaning that the cancer was not getting worse. Further, no information is given about the tumor’s biology nor can any conclusions be drawn in the absence of a comparator such as enzalutamide alone vs the combination.

Readers are told that “patient care guided by CURATE.AI is currently ongoing.” In other words, by the time anyone is reading the release, the patient could be in remission. Or the cancer may have spread. Or the patient’s condition may not have changed at all. We just don’t know.

Does the news release adequately explain/quantify the harms of the intervention?

Not Satisfactory

AI is not perfect. AI programs can, and will, make mistakes. What might those mistakes mean in the context of cancer treatment? It’s not clear, and the release doesn’t address this. We know that clinical trials using “CURATE.AI-guided” therapies are underway, and that the patient highlighted in the news release is involved in a clinical trial. But we don’t know how any of those trials are going. Will those trials identify any risks or potential harms? How will medication dosages be adjusted to reduce side effects? That remains to be seen, and the release should not simply ignore the issue. If this single patient’s treatment was sufficient to warrant a news release, any potential harms (or even the absence of potential harms) should also be worth addressing.

Does the news release seem to grasp the quality of the evidence?

Not Satisfactory

The release makes clear that it is focused only on one patient’s treatment. But that’s not good enough. The release should make clear that, while this patient’s results were promising, there are severe limitations on the extent to which the patient’s results can be extrapolated to a larger population. Instead, the release does just the opposite, calling a single case a “big step forward.” In addition, as noted above under “Explain Harms,” this patient was part of a clinical trial, and there are other clinical trials under way. Are they phase 1 trials? Phase 2? Phase 3? How many patients are involved? What are the study designs? The release raises more questions than it answers.

Does the news release commit disease-mongering?

Not Applicable

No disease mongering here. Some context on the prevalence of prostate cancer would have been useful.

 

Does the news release identify funding sources & disclose conflicts of interest?

Not Satisfactory

It is not made clear who funded the relevant study, who funded the creation of CURATE.AI, or whether CURATE.AI has been patented or is under consideration for being marketed as a for-profit medical tool.

Does the news release compare the new approach with existing alternatives?

Not Satisfactory

CURATE.AI is far from the only AI tool that has been considered for use in cancer treatment. For example, HealthNewsReview.org wrote last year about a (failed) attempt to incorporate IBM’s Watson system into cancer treatment at M.D. Anderson Cancer Center. And there are hundreds of papers that address various attempts to use AI, machine learning, or artificial neural networks in the context of cancer treatment. Here’s a review paper from 2006 that addresses 396 related papers that had already been published at that time. The release on CURATE.AI does not make clear how this new AI tool is different from or similar to the body of work in the field.  We don’t expect a news release to offer a detailed overview of how the new work differs from what came before it, but we do expect a release to provide at least a broad idea of what sets the new work apart.

Does the news release establish the availability of the treatment/test/product/procedure?

Not Satisfactory

The release does make clear that clinical trials are under way. That’s good. However, the release is written as if the clinical trials have already been completed successfully. For example, the release states that “CURATE.AI is applicable to all diseases and all patients.” There is, as far as can be discerned from everything else in the release, nothing that supports that statement. And despite the sweeping language used in the release, it is not at all clear when CURATE.AI may be available for clinical use.

Does the news release establish the true novelty of the approach?

Satisfactory

The release says that “a first generation of the [CURATE.AI] platform was previously validated in the clinic for single drug optimisation,” and that this single-patient study “demonstrates that CURATE.AI can optimise multi-drug regimens.” That’s enough to earn it a satisfactory rating here, but we’ll revisit that language under “Unjustifiable Language.”

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Not Satisfactory

The release uses unsatisfactory language throughout. We’ll provide two examples here. The release tells readers that this study “demonstrates that CURATE.AI can optimise multi-drug regimens.” No, it doesn’t. This study focuses on the experience of one patient. Did the use of CURATE.AI improve the patient’s treatment? It’s impossible to tell — because it is one patient. We simply cannot determine what the cancer’s progression would have been, since we can only the see the data on one individual. The release also states that “CURATE.AI is applicable to all diseases and all patients” — a claim that the release (and related study) cannot support in any way.

Total Score: 1 of 9 Satisfactory

Comments

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.