I highly recommend you read the entirety of Christie Aschwanden’s excellent FiveThirtyEight piece chronicling the problems with many nutrition studies. She takes you inside the production of such a study to show you how the sausage is made (and measured) — and the results aren’t pretty. She writes:
The problem begins with a lack of consensus on what makes a diet healthy. Is the aim to make you slender? To build muscles? To keep your bones strong? Or to prevent heart attacks or cancer or keep dementia at bay? Whatever you’re worried about, there’s no shortage of diets or foods purported to help you. Linking dietary habits and individual foods to health factors is easy — ridiculously so — as you’ll soon see from the little experiment we conducted.
But if you can’t spare the time to delve into Aschwanden’s 3,000-word description of that experiment, here’s the Cliff Notes listicle version of the problems she ran into. These all happen to be points we routinely raise in our reviews of news coverage of nutrition studies.
1. Questionnaire-based nutrition data are inaccurate
After trying to calculate her nutrition intake with the questionnaires used by researchers, Aschwanden found the logistics to be maddeningly difficult. For example:
Some questions — how often do you drink coffee? — were straightforward. Others confounded us. Take tomatoes. How often do I eat those in a six-month period? In September, when my garden is overflowing with them, I eat cherry tomatoes like a child devours candy. I might also eat two or three big purple Cherokees drizzled with balsamic and olive oil per day. But I can go November until July without eating a single fresh tomato. So how do I answer the question?”
2. “Positive” results are often false-positives
The problems with food questionnaires go even deeper. They aren’t just unreliable, they also produce huge data sets with many, many variables. The resulting cornucopia of possible variable combinations makes it easy to p-hack your way to sexy (and false) results, as we learned when we invited readers to take [a questionnaire] and answer a few other questions about themselves. We ended up with 54 complete responses and then looked for associations — much as researchers look for links between foods and dreaded diseases. It was silly easy to find them.
3. Correlation doesn’t equal causation
Our experiment found that people who trim the fat from their steaks were more likely to be atheists than those who ate the fat that god had provided for them. It’s possible that there’s a real correlation between cutting the fat from meat and being an atheist, Vieland said, but that doesn’t mean that it’s a causal one.
4. Benefits are overstated through reporting of relative risks
Relative risks are almost always much more extreme than absolute risk, but absolute risk (your risk of getting cancer if you consume bacon, for instance) is what we really care about. If, say, 1 out of 10,000 people who ate the most bacon got cancer, compared with 3 out of 10,000 who ate none, that’s a threefold difference. But the difference in absolute risk — a 0.01 percent chance of cancer versus 0.03 percent — is tiny and probably not enough to change anyone’s eating habits.
Why do we pay so much attention to research that has so much uncertainty baked into it? Aschwanden speculates that it has something to do with a fundamental human desire for control: “The natural reaction when someone has a heart attack or is diagnosed with cancer is to look for a way to protect yourself from a similar fate. So we turn to food to regain a modicum of control. We can’t direct what’s going on inside our cells, but we can control what we put into our bodies.”
Aschwanden’s report suggests that in many cases, the control we seek through consumption of “good” foods and avoidance of “bad” foods is an illusion. Here at HealthNewsReview.org, we think that better reporting on study limitations can help the public avoid such magical thinking.
Kevin Lomangino is managing editor of HealthNewsReview.org and was formerly editor of Clinical Nutrition Insight.