The biggest shortcoming with this story is the failure to evaluate the quality of the evidence at this point. The story doesn’t emphasize for readers how preliminary this research is. For instance, we’re not told that:
* Research presented at a conference has not been peer-reviewed and is considered preliminary;
* 12 people in a study is tiny;
* There are limitations with what you can say given the extreme short-term length of the study (2 nights);
* Overall, the technology has only been tested at night and not in a home-based, real-world scenario; see the MedPageToday coverage for such caveats;
* This evidence isn’t of such overwhelming quality that no further research is necessary to prove it works: the investigators are planning future studies in the home setting;
* Challenges remain, as pointed out in a recent Lancet editorial, in getting the approach and algorithms to work during the daytime, when there will be varying meals and periods of physical activity.
The results of 12 people using a new technology for 1 night presented without peer review — the extremely early phase of this research simply isn’t conveyed to readers.
Costs aren’t mentioned for the approach being researched nor for its constituent technologies, continuous glucose monitoring and insulin pumps. The two components are not cheap and many insurance companies aren’t covering them except in special circumstances.
We do receive one set of data on benefits, the 70% vs. 47%. We’re unable to verify these numbers, as the abstract available online and MedPageToday coverage both cite "Time in Target" as 72% and 50%. Perhaps 70% and 47% come from a different source, such as the ADA symposium, press briefing, or teleconference with the investigators.
Beyond the numbers, we take some issues with the presentation of benefits. First, the language in the opening paragraph that "an ‘artificial pancreas’…will help better control the swings of blood glucose that come with the disease." That’s a bit too definitive at this stage.
Second, any comparative benefits are not meaningful if we don’t know who the comparator group was. Finally, as we discussed in detail in our review of the HealthDay article, we feel than an independent voice should’ve been provided to tell us the meaning of the benefits. What does the percent of time spent in or out of target range mean? What are the consequences?
Specific data on harms aren’t mentioned, just a mention of a prior study that demonstrated that this approach could reduce dangerous hypoglycemic episodes in children, and the funder’s quote that more and more studies are telling us the approach can be done safely.
Even within the limited scope of this preliminary study, the author could’ve mentioned the data on severe hypoglycemia in these adults (1 episode in the comparator group, 0 for the closed-loop system). That’s harder to do when the article doesn’t name the comparator therapy. The story also could have highlighted the challenges of using this technology even in a controlled research setting.
The author neglects to tell her lay readers, in several ways, how preliminary this research is. For instance, we’re not told that:
* Research presented at a conference has not been peer-reviewed and is considered preliminary
* 12 people is not enough to play softball, no matter prove the efficacy, safety, and inevitable marketing of a new therapy; the author didn’t have to get into the nitty-gritty of statistical power but simply acknowledge that 12 is tiny
* The study was only for 2 nights, and only for 1 night were subjects actually using the new technology (vs. the comparator technology); it’s extremely short-term
* Overall, the technology has only been tested at night and not in a home-based, real-world scenario; see the MedPageToday coverage for such caveats
* This evidence isn’t of such overwhelming quality that no further research is necessary to prove it works: the investigators are planning future studies in the home setting
* The study focused on the overnight period, which required subjects to separately take insulin with their meal before using the device; so the "artificial pancreas" didn’t replace mealtime insulin
* Challenges remain, as pointed out in a recent Lancet editorial, in getting the approach and algorithms to work during the daytime, when there will be varying meals and periods of physical activity
We’re not naysayers or mindless contrarians. We want this research to work fantastically well and become available to patients soon. But we also want readers to understand that many investigational therapies at the same stage as this technology fail, or even prove more harmful than helpful, when they undergo more advanced research and never make it to patients. Hope doesn’t have to be false, and optimism can be cautious.
Some other more technical details about the evidence are absent. It’s vague about the comparator group, saying it was the time when subjects "didn’t use the technology." It should be clear, the technology here is the computer algorithm and researcher manual adjustment of the pump; the approach was compared to conventional insulin pump therapy, and that’s what the 47% refers to. (As this was a crossover study, the subjects had 1 night of each approach.)
See our comments on the other criteria, such as Availability, for more gaps and sinkholes in this story.
The article doesn’t overstate the significance of diabetes.
One presumably unaffiliated source may have been consulted about the study. We say "may" because the paragraph is vague:
Even with this technology, you’re still working backward, says endocrinologist Robert Rizza, executive dean of research at the Mayo Clinic. "It takes time for that insulin to be absorbed. Despite that fact, it’s been working very well and helps avoid very high and very low blood sugars."
When he says "it’s been working very well," is he offering independent evaluation of this research on closed loop insulin delivery? Or does he mean any insulin pump — which falls under the same description — has in the real world been "working very well and helps to avoid very high and very low blood sugars" ? Being this vague, it’s hard for us to say we got an independent take on this research, particularly with the lack of caveats to be found anywhere on the preliminary nature of the research.
The article doesn’t compare the approach with available alternatives, not even the existing alternative the study itself used as a comparator group. When the authors summarized the Lancet study from earlier this year, which showed that this approach "could better control blood glucose levels during sleep and reduce the chance of hypoglycemia," we’re left with the question — compared to what? We’re not disputing the summary of that study, but what did the study compare the artificial pancreas to?
While the article is clear that closed-loop glucose control is not currently available, it makes availability seem inevitable. The opening line says that scientists are "getting closer to offering" it. A representative from a funder says the group believes the technology could be available within the next few years." A non-peer-reviewed study of 12 subjects over 2 nights does not guarantee availability, and it’s inappropriate to assume it.
According to the study’s abstract, the approach required manual adjustments of insulin infusion rate. We take that phrase to mean that the "artificial pancreas" in this study isn’t a device, like a prosthetic pancreas that might be implied by the catchy name. Rather, the study required researchers to manually adjust the patient’s insulin pump while the patient slept (checking every 15 minutes), as opposed to that autonomous feedback loop between monitor and pump described in the study, like an algorithm that automatically and safely adjusts the infusion rate based on the data from the continuous glucose monitor. That’s why this research is still "in clinic." And, unless this artificial pancreas will come with a team of clinically trained elves who stay up all night to operate the pump, it shows that the approach is not as ready for prime-time as a lay reader would think from the article and its definition of a device-like "artificial pancreas."
The article is clear which devices within the "artificial pancreas" approach are already in use and the novelty of the computer program that helps the two devices talk to each other.
As noted, the story could’ve been clearer about the difference between that system and the preliminary one used in the study, which required manual adjustments to the pump overnight — not just sophisticated software — to help the glucose monitor and insulin pump chat.
Not applicable because we don’t know the sources used in the study. It doesn’t appear to rely overly on the ADA press release. However, several stories in the media that covered both the Hovorka study and the Ritholz study explicitly relied heavily on a press briefing or teleconference, and the flow leads us to think this story was no different.