NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine - https://archive.org/web/.
Read Original Release

With second death-risk release in a month, Lancet displays better grip on limitations

Rating

5 Star

Simple score predicts risk of death for middle-aged adults in the UK

Our Review Summary

crystal ballFor the second time in a month, The Lancet is promoting a “simple” test that may help predict one’s risk of death. A few weeks ago it was the notorious death grip study, this time it’s a questionnaire that predicts your “Ubble age.”

The new test is based on Swedish researchers’ efforts to correlate an array of lifestyle and health factors with mortality in a massive epidemiological study of UK residents. They are making the test publicly available via an interactive website with the help of the UK organization Sense About Science, which tries to help citizens utilize scientific evidence in their daily lives. [Editor’s note: We could not log onto the death risk calculator website to evaluate it despite multiple attempts as of this writing.]

While these tests are never as “simple” as they’re made out to be, we’re happy to note significant improvement in this most recent release compared with its predecessor. The current release does a pretty thorough job of explaining the study and quotes from an accompanying editorial to provide some perspective. The editorial notes that it’s not clear how individuals may react to such estimates of their death risk. Is this information motivating to people or not? And if it is motivating, is it healthy motivation that leads to actions that decrease future risk, or does it engender a sense of hopelessness that leads the person to take on even more unhealthy behaviors resulting in even higher risk? We applaud the release for calling attention to this important unanswered question. But we do wonder if more thought should have been given to the answer before trumpeting the results to the world and making the test freely accessible to everyone.

 

Why This Matters

Predicting what the future may hold is something that is intriguing to everyone. And what is more important to predict than whether you will live or die? So this algorithm that provides an estimate of one’s risk of death in 5 years is appealing at many levels. However, there are several important caveats — some addressed in this news release and some not. The biggest missing piece is whether the information is accurate for an individual. While the study reports reasonable population accuracy (~80%), there is some mention that it varies. For example, the test does a better job for younger individuals than for older ones. However, there is no information provided about how accurate the individual’s estimate is. Without such information, individuals should be very cautious about interpreting data that may be accurate only at a population level until more information is available for their particular situation.

Criteria

Does the news release adequately discuss the costs of the intervention?

Not Applicable

Access to an interactive site for purposes of computing a score predicting one’s risk of dying will be free. However, this should have been stated in the release. We can’t rate the release Satisfactory if there’s no mention of cost at all, so we’ll rate it Not Applicable.

Does the news release adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

Very little about this news release is quantitative. The document repeatedly lauds the benefits of predicting 5-year mortality via self-reports rather than more intrusive physical tests, and it touts the potential use of the “Ubble age” score to assist doctors in identifying “high risk” patients and to improve individuals’ self-awareness. But no studies have been conducted to test effects on either of these audiences, a concern which the release refers to only at the end of the document. The release could have included that information higher up in the release and emphasized it more.

The release does mention that the algorithm was validated and shown to have an ~80% accuracy. But as noted above, this is a population level estimate. The overall test characteristics of the model do not translate into an interpretation that is meaningful for an individual. One could argue that this is mainly the fault of the original study rather than the release, as we could find no data in the paper that provides information that an individual could use to determine how accurate their particular calculation was. However, this problem contributes to an overall sense that the release is weighted too far in the direction of benefits without enough attention to potential harms and uncertainties.

Does the news release adequately explain/quantify the harms of the intervention?

Satisfactory

The text does offer a brief cautionary note when it quotes from the commentary of a couple of UK scientists in The Lancet issue to the effect that “whether this will help individuals improve self-awareness of their health status…or only lead to so-called cyberchondria, is a moot point.” Cyberchondria refers to unfounded anxiety concerning the state of one’s health brought on by visiting health and medical websites, so that term seems apt here. And “moot” conveys the uncertainty surrounding outcomes as, again, the study can offer no evidence. We’ll give a Satisfactory rating, though we wonder if it would have been more effective to say, “it is unclear whether use of this information will lead to actions that will help or harm individuals.”

Does the news release seem to grasp the quality of the evidence?

Satisfactory

The data used came from a gargantuan epidemiological study involving nearly half a million UK respondents. The news release offers some details about that study, including the number of possible predictors (655) and the nature of the analysis. The text also contains a caveat reminding the reader that prediction is not the same as causation (a variant on the “correlation is not causation” theme), as well as a quote from one of the investigators that cautions against seeing one’s score as a “deterministic prediction.” These are important qualifiers, but they occur lower in the press release, and it is not clear how well those points will get picked up in subsequent stories.

Does the news release commit disease-mongering?

Satisfactory

We didn’t see anything that implied a doomsday perspective to use of this data. The release also clearly implies this is for UK citizens and not for others. So we’ll award a Satisfactory rating, although we’d note again that there’s an aspect of this test that could possibly lead to increased anxiety and “cyberchondria,” as the release puts it.

Does the news release identify funding sources & disclose conflicts of interest?

Satisfactory

Funders are clearly identified in the notes at the bottom of the press release.  The investigators assert the absence of conflicts of interest in the journal article itself.

Does the news release compare the new approach with existing alternatives?

Satisfactory

The study and its press release make much of the potential advantage of predicting death via self-report questions rather than biological tests. So the text makes an overt comparison, admittedly without data.

The release could also have included the comparison to doing nothing. Whether providing any risk information (this version or another) or no risk information makes a difference is a legitimate question.

Does the news release establish the availability of the treatment/test/product/procedure?

Satisfactory

Notes at the end of the news release speak to the availability of the interactive “Ubble” website once the research article is published. We could not access the calculator, perhaps due to high traffic levels accompanying the rollout.

Does the news release establish the true novelty of the approach?

Satisfactory

The news release is explicit about the novelty of the approach to prediction, as well as regarding the scale of the study itself. We suspect it is correct in this assessment, as the epidemiological study on which the work is based is truly huge.

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Satisfactory

One of the investigators refers to the study’s development of a prediction score from self-report data as “exciting,” but most of the quotes, while positive, also convey useful information, including qualifiers.

Total Score: 8 of 9 Satisfactory

Comments (1)

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.

Laurence Alter

June 16, 2015 at 3:21 am

Dear Staff:
Thank you for reviewing such top-tier publications like “The Lancet”; you might also consider “Science” and “Nature” [United Kingdom journals] when they have applicable medical articles.
Respectfully,
Laurence

Reply