There’s no question that this story looked thoroughly at the evidence being presented. The story makes many legitimate points about limitations in the study and why it should be interpreted very cautiously. Even if some may feel that the story skirts the line, we’ll gladly choose this story’s aggressive posture over the credulous “shovelware” that often passes for health journalism these days.
While we applaud the aggressiveness with which some stories like this one punched holes in this study, we pause to point out that many of the complaints being made here — marginal benefit, lots of dropouts, possibly biased investigators, etc., etc. — can also be made about studies of more “mainstream” treatments. We will continue to push journalists to call out those studies as aggressively as they do complementary or alternative health research.
The story says that chelation therapy costs about $5000 out of pocket.
The story says the benefits were “very small, with 26.5% of those randomly assigned to chelation experiencing a cardiovascular problem, compared with 30% of those randomly assigned to a placebo infusion.” That’s a satisfactory handling of the statistics.
The story could have also pointed out that the primary outcome was a so-called “composite endpoint” encompassing a lot of different results, including death, heart attack, stroke, referral for cardiovascular procedures, and others. Most of the benefit attributed to chelation was due to fewer cardiovascular procedures being performed in that group, which is an important detail. Because this outcome involves some subjective determination about who should and who should not be referred for these procedures, it is considered a “softer” or less reliable outcome than rates of death or heart attack, which are outside of the investigators’ control. If chelation provided a substantial benefit, we presumably would have seen a larger difference on these other “harder” endpoints.
We’re also a little uneasy with the story’s discussion of statistical significance, which it says is “the standard for judging whether a result is real or a fluke.” It says it was “concerning” that the results of this study passed the significance bar “by only 0.001,” implying that the results would have been considered a “fluke” if the results were a few hundredths different. In fact, the results, taken at face value, suggest that there was about a 97% probability that this result could not have occurred by chance, and a change of .001 would not have altered that fact appreciably — even if it did mean that the findings would no longer be considered “statistically significant.” We think it’s a stretch to question this result — something which is almost never done in coverage of other studies that also “barely clear the bar” for significance — without a more complete discussion of what this means.
The story says that chelation is risky and dangerous, and notes that at least 30 patients have died from off-label chelation therapy since the 1970s. But then it quotes the director of the National Heart, Lung, and Blood Institute calling the study a “breakthrough” – not for proving that chelation works, but for “establishing that this chelation therapy regimen can be administered in a safe manner in the context of a clinical trial.” Without some data or an explanation, readers are going to have a hard time reconciling these contradictory views about the safety of chelation. And since the NHLBI quote ends the piece, it’s the take-home that may have the biggest influence on readers.
Mixed bag on this criterion, but we’ll give it the benefit of the doubt.
The story included repeated deserved criticism of the study and how it was conducted. Among other problems, the story notes that the study had high drop out rates – a factor that can introduce bias. And the story mentioned that patients had differing baseline cholesterol levels, which also could have led to a skewed finding favoring chelation.
2nd last line in the story: “All of these factors call into question the results,” says Michael Carome, deputy director of the advocacy group Public Citizen’s Health Research Group.
On the other hand, there was no warning that the findings have only been presented at a medical conference and haven’t been thoroughly peer-reviewed for publication in a journal; we may learn a lot more about how the study was conducted, and what the results mean, after the paper has been vetted by other experts as part of this process.
The story gets a satisfactory score for this criterion.
You could argue that the study wandered into fear-mongering territory in its discussion of chelation. There’s no doubt that chelation can be a very dangerous procedure in the wrong hands, but the primary result here seems to be that it was safely administered in the context of the study. It’s not until the last sentence–and after chelation is repeatedly characterized as dangerous and possibly deadly — that the story makes this point.
Then again, this study seems so riddled with problems that it’s hard to know what to make of the results. We’ll give the benefit of the doubt here, as there’s certainly no reason to recommend chelation, and still plenty of reason to be worried about it.
Plenty of independent sources contribute to this piece. If there’s one thing we question, it’s the description of possible bias among the investigators. The story quotes a source who says that “more than half the trial sites were led by doctors who practice chelation,” which “could lead to a bias in favor of chelation.” That may well be true, but aren’t a lot of trials led by physicians and surgeons who use the treatment that they’re supposed to be studying–whether it’s a drug or surgical procedure? We agree that this is a form of bias, but it’s one that’s endemic to medical research. It feels like a double standard to flag the study for this particular shortcoming.
The story mentions that there are medications and lifestyle changes that clearly have been shown to reduce heart disease. That’s good enough.
Another mixed bag for which we’ll give a satisfactory score.
The story is not clear about exactly what chelating agent was used in this study, or where one would go to find a practitioner who can prescribe it. (But, to its credit, it mentioned problems with off-label prescribing.) It was relatively easy to find out online that EDTA was the agent used in the study.
We give the story for credit for reporting that “the Food and Drug Administration had taken the study drug off of its list of approved medications.”
This is the only large-scale trial of chelation for prevention of heart disease. The research is quite novel.
There is enough original reporting for us to be sure this story wasn’t based on a press release.
Systematic, Criteria-Driven
Monday at #ADA2021, an infectious disease epidemiologist, a journalist, and a researcher shared perspectives on communication of risk in the era of COVID-19: http://ow.ly/ROfR30rMeE7 @AstleyCM @garyschwitzer @berthahidalgo @ADA_DiabetesPro
Shoutouts to @susan_bewley @drkevinknopf @MichaelBaum11 in this call for more than fawning coverage from press releases for these kinds of stories.
Also, linked history inside goes back 5 years just on Grail apparently trying to become Holy Grail. https://twitter.com/garyschwitzer/status/1408849696416841731
Comments
Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.
Our Comments Policy
But before leaving a comment, please review these notes about our policy.
You are responsible for any comments you leave on this site.
This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.
We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.
We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.
There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”
The authors of the Retraction Watch comments policy urge commenters:
We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.
And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.
The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.
You might also like