Kudos to journalists who evaluate the quality of the evidence

The Los Angeles Times and the Wall Street Journal deserve credit for pointing out the limitations of some types of studies.

The Times reported on a study questioning whether the dietary supplement quercetin lived up to its hype that it might improve athletic performance. Key quote in the story:

The results were disappointing, said the lead author of the paper, Kirk Cureton, a kinesiologist at the University of Georgia in Athens. “Our hypothesis, based on previous studies in mice, was that we would see positive effects. But our findings are important because they suggest that results from the animal studies shouldn’t be generalized to humans,” Cureton said in a news release.

And the Journal wraps up eight recent studies, commenting, on one or another:

• Because it used a one-time survey, originally intended to track breast cancer, the study’s researchers were unable to follow patients as their diagnoses changed over time, and thus couldn’t conclude that obesity or abdominal fat definitively led to asthma.

• The study relied on parental reporting to determine when the infants reached these motor milestones. If enough parents deliberately or unintentionally avoided reporting delayed development, the findings could be skewed, the researchers said.

• The study was too small, the researchers said, to conclude whether immediate treatment of these patients prevented heart attacks, death, or other long-term adverse effects.

• The study was conducted on mice only. More research is required before such findings can be generalized to the management and erasure of human fears.

• The study also included white men, black men and black women, but the researchers found a strong association between PKNOX2 and multiple-drug addiction only for the 1,393 white women in the study, which has 3,627 subjects. They suggested, however, that different genes could play a similar role for these other subjects.

• The study didn’t measure the years of experience, hours since sleeping, or any other characteristic of the surgeons, making it difficult to determine the underlying reasons for the after-hours patients’ poor outcomes.

• The study excluded patients with high social anxiety, which is thought to increase the time required to process facial expressions and involve a different cognitive pathway, the researchers said.

• The study was small–just 275 subjects–and 40% of the patients dropped out before the end of the three-month study (most of them because their symptoms became worse during the trial).

Following my criticism of the BMJ last week for failing to point out the limitations of observational studies and failing to explain that association does not equal causation, it is refreshing to see journalism examples like these that show how it should be done.