NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine - https://archive.org/web/.
Read Original Release

Rockefeller PR release on automated melanoma screening downplayed false-positive rate

Researchers develop automated melanoma detector for skin cancer screening

Our Review Summary

This news release reports on a study of a computerized technology that combines imaging with digital analysis to help improve the screening of melanoma skin lesions.

The news release explains the specificity and sensitivity of the test. It also clearly discloses who funded the research. One downside is that it’s not until the final paragraph that we learn this is still experimental research that must be confirmed in larger studies. Earlier statements make it sound farther along, e.g., “researchers at The Rockefeller University have developed an automated technology that combines imaging with digital analysis and machine learning to help physicians detect melanoma at its early stages.”

A larger concern is how the release downplayed the high-false positive rate of 64%, saying this “approaches” the false-positive rate of manual detection by expert dermatologist, which studies put at about 32%.

 

Why This Matters

Melanoma can be very difficult to diagnose so if doctors could use a tool that helped better screen suspicious-looking moles, they might be able to improve the accuracy of diagnosis and save lives. This new approach thus far has shown promise, but further research will be needed to validate it as an enhanced diagnostic tool–and that should be made upfront and clear to the journalists and consumers who read the news release.

Criteria

Does the news release adequately discuss the costs of the intervention?

Not Applicable

This technology seems to still be in the experimental stage.

Does the news release adequately quantify the benefits of the treatment/test/product/procedure?

Satisfactory

The news release reports a 98% sensitivity and 36% specificity for melanoma detection, which is a level that approaches that of an expert lesion diagnosis.

But the release could have noted that this still isn’t very accurate: A 36% specificity means that 64% of those without melanoma will test positive (i.e. false positive). That translates to a lot of unnecessary anxiety and follow-up testing to rule out suspicious growths.

Does the news release adequately explain/quantify the harms of the intervention?

Not Satisfactory

The release says the test’s ability to diagnose normal moles was 36 percent, “approaching the levels achieved by expert dermatologists performing visual examinations of suspect moles under the microscope.”

But as noted above, that statement downplays the fact that the test will result in a very high number of false-positives. A 36% specificity means that nearly two thirds of non-malignant moles will test positive and require additional follow-up. A test that correctly identifies nearly all melanomas, but also flags more than half of normal growths as melanomas, isn’t all that useful.

Does the news release seem to grasp the quality of the evidence?

Not Satisfactory

The investigators note that this research needs to be confirmed with larger studies–but not until the end of the release. More should have been included on this point, and higher up, so as to not mislead readers into thinking this product is coming to market soon.

And less needed to be said on the potential applications, which is still in “hearsay” phase considering the state of research.

For example, this quote: “I think this technology could help detect the disease earlier, which could save lives, and avoid unnecessary biopsies too,” is premature.

Does the news release commit disease-mongering?

Satisfactory

No obvious disease mongering is evident here.

Does the news release identify funding sources & disclose conflicts of interest?

Satisfactory

We are told that “this work was supported in part by the National Institutes of Health and in part by the Paul and Irma Milstein Family Foundation and the American Skin Association.”

Does the news release compare the new approach with existing alternatives?

Not Satisfactory

The release says “the ability of the test to correctly diagnose normal moles was 36 percent, approaching the levels achieved by expert dermatologists performing visual examinations of suspect moles under the microscope.”

That differs from what’s in the study, which reports “the visual examination of pigmented lesions by expert dermatologists using dermoscopy and following criteria such as the Menzies (or CASH[3]) method has yielded diagnostic accuracy as high as 98% sensitivity and 68% specificity in some studies.”

Sixty-eight percent specificity is much more accurate than 36%, so we’re not sure how that can be defined as “approaching the levels” of accuracy of visual exams by expert dermatologist.

Does the news release establish the availability of the treatment/test/product/procedure?

Not Satisfactory

Some of the language used suggests the technology is mature, “Now, researchers at The Rockefeller University have developed an automated technology that combines imaging with digital analysis and machine learning to help physicians detect melanoma at its early stages.”

It is not until the final paragraph that the need for further research is noted.

Does the news release establish the true novelty of the approach?

Satisfactory

The release establishes that this is a potential new way to augment melanoma detection.

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Satisfactory

The release does not use sensational language.

However, we did want to highlight this quote, which was problematic:

“The success of the Q-score in predicting melanoma is a marked improvement over competing technologies,” says Daniel Gareau, first author of the report and instructor in clinical investigation in the Krueger laboratory.”

We think that it is a bit premature to be this definitive.  A recent review of competing technologies suggests that confocal scanning laser microscopy has a sensitivity of 88-98% and a specificity of 83-99%.

Total Score: 5 of 9 Satisfactory

Comments

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.