NOTE TO READERS: When this project lost substantial funding at the end of 2018, I lost the ability to continue publishing criteria-driven news story reviews and PR news release reviews - once the bread-and-butter of the site going back to 2006. The 3,200 archived reviews, while still educational, are getting old and difficult for me to technically maintain on the back end of the website. So I am announcing that I plan to remove these reviews from the site by April 1, 2021. The blog and the toolkit - two of the most popular features on the site - will remain. If you wish to peruse the reviews before they disappear, please do so by the end of March 2021. After that date you may still be able to access them via the Internet Archive Wayback Machine - https://archive.org/web/.
Read Original Story

NPR suggests experimental concussion ‘spit test’ can improve care, but that’s not what researchers looked at

Rating

3 Star

Categories

Spit Test May Reveal The Severity Of A Child's Concussion

Our Review Summary

This story about a saliva test that may be able to more accurately predict how long children will feel the symptoms of a concussion confuses readers with claims that the test could improve concussion care, something the study did not look at.

We applaud the story for pointing out that the lead researcher is consulting with a company that hopes to market the test. The story also included an independent source who provided some context; however, the way those comments were reported makes it appear the test could detect brain injuries that don’t produce symptoms, when actually the study looked only at how well the test predicted the duration of symptoms.

The story also presents statistics in a way that is technically correct, but probably misleading, because it does not explain how the prevalence of a condition affects the likely meaning of test results. (Specifically, the less common a condition is, the less likely it is that a positive test result is accurate.) And as with all stories about tests, the potential harm of false positive and false negative results should have been mentioned.

 

Why This Matters

News stories about tests should be clear about the questions a test is capable of answering. In this case, the results of a small study indicate that a saliva test might be better than a commonly-used survey at predicting how long a child will report symptoms after a concussion. But the story, especially comments from an independent source, gives the impression that the study showed this test could improve care, in part by detecting brain injuries that don’t produce symptoms.

While researchers are indeed looking into whether biomarkers in saliva or blood could provide new insights to brain injuries, this particular study involved only predictions about symptom duration, there were no data presented about the severity or proper care of the underlying brain trauma. News stories should be scrupulous about distinctions between what researchers hope to prove and what they actually tested.

Criteria

Does the story adequately discuss the costs of the intervention?

Not Satisfactory

Although this research is in early stages and it could be argued that it is too early to insist on a discussion of cost, the eventual price of this test will determine whether it would be used widely or not. For example, if it cost only a few dollars, it might be widely used by pediatricians (assuming it really works), but if it cost hundreds of dollars, it might not be used. The story should have at least tried to include some sort of ballpark cost estimate in order to tell the audience whether this sort of test would be practical.

Does the story adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The story attempts to quantify benefits with this statement:

A test that measures fragments of genetic material in saliva was nearly 90 percent accurate in identifying children and adolescents whose symptoms persisted for at least a month, a Penn State team told the Pediatric Academic Societies Meeting in San Francisco, Calif.

The story overstates both the accuracy of the test and the meaning of the study results. First, regarding the accuracy, while it is technically correct that the researchers reported their spit test “was nearly 90 percent accurate in identifying children and adolescents whose symptoms persisted for at least a month,” that statement is likely to give most people an inflated sense of what test results would mean in the real world.

The researchers reported in their conference abstract (see session ID 255639) that the test had a sensitivity (ability to detect true cases) of 90 percent and specificity (ability to rule out non-cases) of 87 percent. But the lead researcher estimates that only 25 percent of children have prolonged symptoms after a concussion. Putting those numbers into a statistical calculator indicates that in a typical group of patients, a positive result would mean the likelihood of prolonged symptoms would be actually less than 70 percent. Still, a negative result would be correct more than 96 percent of the time.

Also, where is the evidence that predicting symptom duration improves outcomes? Would the test result change treatment? Would it alter decisions about when it is safe for a child to return to sports? Of course, parents want to know “’When is my child going to be back to normal again?,’” but most readers would infer that the test must have some value beyond making predictions that don’t change how the child is actually treated.

What’s more, the story includes a claim that doesn’t make sense based on the evidence the researchers presented. It paraphrases an expert who says “A saliva test could greatly improve care for young people who don’t have obvious symptoms of a concussion.” But that’s not what the researchers looked at  They reported that the saliva test was able to predict the duration of symptoms, which doesn’t reveal anything about its ability to detect brain damage that does not produce symptoms. That is an important question that still has to be tested.

Does the story adequately explain/quantify the harms of the intervention?

Not Satisfactory

There is no discussion of potential harms, which would include false positive and false negative results. For example, if a child’s test result erroneously said he would have prolonged symptoms (a false positive result), he may miss out on school or activities, for no good reason. And if the result instead erroneously said the child wouldn’t have prolonged symptoms (a false negative result), the child may resume school or activities too early.

Does the story seem to grasp the quality of the evidence?

Not Satisfactory

The story did not adequately discuss the quality of the evidence–this was a small group of patients, just 50 patients, of a broad age range of children under age 18. This small patient sample can affect the statistical significance of the results–more research is needed.

Also, these study findings have not been published in a peer-reviewed journal, so they are considered preliminary. This should have been noted as well.

Does the story commit disease-mongering?

Satisfactory

The story does not exaggerate the seriousness of concussions.

Does the story use independent sources and identify conflicts of interest?

Satisfactory

The story reports that the lead researcher “helped develop the test and consults for a company that hopes to market concussion tests.” It also includes comments from a source who does not appear to be connected with the study, nor have any potential conflicts of interest. Though as we noted above, the way this source’s comments are reported creates confusion about what the study actually looked at.

Does the story compare the new approach with existing alternatives?

Satisfactory

The story does compare the spit test to a widely used concussion survey. However, many readers and listeners are likely to be confused about the basis of the comparison, and incorrectly believe that the researchers compared how well the tests guide patient care, when actually the new study looked only at the ability to predict reports of symptoms.

Does the story establish the availability of the treatment/test/product/procedure?

Satisfactory

It is clear that the spit test is still being studied. There were no predictions about when or if it would be available.

Does the story establish the true novelty of the approach?

Satisfactory

A spit test to predict the duration of concussion symptoms would indeed be new. Also, the description of the background of the independent source in the story notes that he is also working on using microRNA tests to better understand traumatic brain injury, so readers and listeners are told that other researchers are studying similar approaches.

Does the story appear to rely solely or largely on a news release?

Satisfactory

There are quotes in the story that indicate original reporting was done.

Total Score: 6 of 10 Satisfactory

Comments

Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.