This story about a saliva test that may be able to more accurately predict how long children will feel the symptoms of a concussion confuses readers with claims that the test could improve concussion care, something the study did not look at.
We applaud the story for pointing out that the lead researcher is consulting with a company that hopes to market the test. The story also included an independent source who provided some context; however, the way those comments were reported makes it appear the test could detect brain injuries that don’t produce symptoms, when actually the study looked only at how well the test predicted the duration of symptoms.
The story also presents statistics in a way that is technically correct, but probably misleading, because it does not explain how the prevalence of a condition affects the likely meaning of test results. (Specifically, the less common a condition is, the less likely it is that a positive test result is accurate.) And as with all stories about tests, the potential harm of false positive and false negative results should have been mentioned.
News stories about tests should be clear about the questions a test is capable of answering. In this case, the results of a small study indicate that a saliva test might be better than a commonly-used survey at predicting how long a child will report symptoms after a concussion. But the story, especially comments from an independent source, gives the impression that the study showed this test could improve care, in part by detecting brain injuries that don’t produce symptoms.
While researchers are indeed looking into whether biomarkers in saliva or blood could provide new insights to brain injuries, this particular study involved only predictions about symptom duration, there were no data presented about the severity or proper care of the underlying brain trauma. News stories should be scrupulous about distinctions between what researchers hope to prove and what they actually tested.
Although this research is in early stages and it could be argued that it is too early to insist on a discussion of cost, the eventual price of this test will determine whether it would be used widely or not. For example, if it cost only a few dollars, it might be widely used by pediatricians (assuming it really works), but if it cost hundreds of dollars, it might not be used. The story should have at least tried to include some sort of ballpark cost estimate in order to tell the audience whether this sort of test would be practical.
The story attempts to quantify benefits with this statement:
A test that measures fragments of genetic material in saliva was nearly 90 percent accurate in identifying children and adolescents whose symptoms persisted for at least a month, a Penn State team told the Pediatric Academic Societies Meeting in San Francisco, Calif.
The story overstates both the accuracy of the test and the meaning of the study results. First, regarding the accuracy, while it is technically correct that the researchers reported their spit test “was nearly 90 percent accurate in identifying children and adolescents whose symptoms persisted for at least a month,” that statement is likely to give most people an inflated sense of what test results would mean in the real world.
The researchers reported in their conference abstract (see session ID 255639) that the test had a sensitivity (ability to detect true cases) of 90 percent and specificity (ability to rule out non-cases) of 87 percent. But the lead researcher estimates that only 25 percent of children have prolonged symptoms after a concussion. Putting those numbers into a statistical calculator indicates that in a typical group of patients, a positive result would mean the likelihood of prolonged symptoms would be actually less than 70 percent. Still, a negative result would be correct more than 96 percent of the time.
Also, where is the evidence that predicting symptom duration improves outcomes? Would the test result change treatment? Would it alter decisions about when it is safe for a child to return to sports? Of course, parents want to know “’When is my child going to be back to normal again?,’” but most readers would infer that the test must have some value beyond making predictions that don’t change how the child is actually treated.
What’s more, the story includes a claim that doesn’t make sense based on the evidence the researchers presented. It paraphrases an expert who says “A saliva test could greatly improve care for young people who don’t have obvious symptoms of a concussion.” But that’s not what the researchers looked at They reported that the saliva test was able to predict the duration of symptoms, which doesn’t reveal anything about its ability to detect brain damage that does not produce symptoms. That is an important question that still has to be tested.
There is no discussion of potential harms, which would include false positive and false negative results. For example, if a child’s test result erroneously said he would have prolonged symptoms (a false positive result), he may miss out on school or activities, for no good reason. And if the result instead erroneously said the child wouldn’t have prolonged symptoms (a false negative result), the child may resume school or activities too early.
The story did not adequately discuss the quality of the evidence–this was a small group of patients, just 50 patients, of a broad age range of children under age 18. This small patient sample can affect the statistical significance of the results–more research is needed.
Also, these study findings have not been published in a peer-reviewed journal, so they are considered preliminary. This should have been noted as well.
The story does not exaggerate the seriousness of concussions.
The story reports that the lead researcher “helped develop the test and consults for a company that hopes to market concussion tests.” It also includes comments from a source who does not appear to be connected with the study, nor have any potential conflicts of interest. Though as we noted above, the way this source’s comments are reported creates confusion about what the study actually looked at.
The story does compare the spit test to a widely used concussion survey. However, many readers and listeners are likely to be confused about the basis of the comparison, and incorrectly believe that the researchers compared how well the tests guide patient care, when actually the new study looked only at the ability to predict reports of symptoms.
It is clear that the spit test is still being studied. There were no predictions about when or if it would be available.
A spit test to predict the duration of concussion symptoms would indeed be new. Also, the description of the background of the independent source in the story notes that he is also working on using microRNA tests to better understand traumatic brain injury, so readers and listeners are told that other researchers are studying similar approaches.
There are quotes in the story that indicate original reporting was done.
Systematic, Criteria-Driven
Like Michael Corleone of science reporting that @garyschwitzer is (he tries to get out, bullshit disguised as #science #journalism keeps pulling him back in!), he drops this on @nytimes idiocy 🙄 in reporting T1 diabetes "cure"
HT @HealthNewsRevu
https://www.healthnewsreview.org/2021/11/cure-for-type-1-diabetes-weve-been-down-this-road-before-with-ny-times/
A great read on this weekend’s NYT claims of a cure for Type 1.
Cure for Type 1 diabetes? We’ve been down this road before with NY Times https://www.healthnewsreview.org/2021/11/cure-for-type-1-diabetes-weve-been-down-this-road-before-with-ny-times/
"But it is the framing and the emphasis of the #journalism by a leading science journalist at a leading newspaper that demands scrutiny. It has an impact on readers and on sick people that may cause harm by promoting false hope about extremely preliminary research"
👍👏💪 truth https://twitter.com/garyschwitzer/status/1465399358137024518
Comments
Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.
Our Comments Policy
But before leaving a comment, please review these notes about our policy.
You are responsible for any comments you leave on this site.
This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.
We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.
We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.
There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”
The authors of the Retraction Watch comments policy urge commenters:
We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.
And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.
The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.
You might also like