In 2017 we will have a regular column highlighting media messages that should have listened to our frequent reminder:
Association ≠ Causation.
In other words, they fail to remind readers and listeners about the clear limitations of observational studies that cannot prove cause and effect. Yet they often use causal language to describe the results of such research. We’ve offered readers a primer on this topic for years.
The big story of the day, erroneously reported in the following examples:
TIME commits the time-honored flaw with its headline, “These three sports will help you live longer.” (Time out: not proven.)
Reuters repeats the error with, “Want to delay death? Then swim, dance, or get on court, study says.” (No, it didn’t.)
The Guardian isn’t guarding against hype with its headline, “Health racquet: Tennis reduces risk of death at any age, study suggests.” (No proof of reduction)
HealthDay hedges its headline with a “may” inserted: “Tennis anyone? It may prolong your life” (Hedging with qualifiers not good enough in our view)
The Los Angeles Times leaped from the observational study to even offering advice: “For a long life, consider picking up a tennis racket.”
As that story explained, “The results, published Tuesday in the British Journal of Sports Medicine, are based on data from 43,705 women and 36,601 men who participated in the Health Survey for England or the Scottish Health Survey. These volunteers, whose average age was 52, told interviewers how often they exercised, what type of exercise they did and how intense their workouts were.”
Whenever you see stories that leap to conclusions based on questionnaires that rely on memory and self-reporting, as this research did, red flags should pop up.
Even the journal news release emphasized the following – which did not appear in any of the stories I saw in my quick scan this morning:
This is an observational study so no firm conclusions can be drawn about cause and effect, added to which the relatively short recall period, the ‘seasonality’ of certain sports, and the inability to track changes in levels of sports participation throughout the monitoring period, may all have had some bearing on the results, caution the researchers.
There will be many other stories about this observational study today. As you read them, remember this: sometimes it makes sense to consider observational data when making public health recommendations. And maybe this is one of those times. But the quality of the evidence is the paramount consideration. And in the examples above I saw very little to no independent journalistic vetting of the quality – and limitations – of this study. It is simply inaccurate for so many journalists to use causal language – with its implication of the certainty of the findings – to describe research that is far from certain. That, in my view, is not only bad editorial judgment. It is an ethical issue because it misleads people. And if you don’t know what the limitations are – or if you do know but choose not to include them – I submit that because you’re misleading people you probably should not report on this type of research at all.