How the sausage inside a nutrition study is made

christiesquareI highly recommend you read the entirety of Christie Aschwanden’s excellent FiveThirtyEight piece chronicling the problems with many nutrition studies. She takes you inside the production of such a study to show you how the sausage is made (and measured) — and the results aren’t pretty. She writes:

The problem begins with a lack of consensus on what makes a diet healthy. Is the aim to make you slender? To build muscles? To keep your bones strong? Or to prevent heart attacks or cancer or keep dementia at bay? Whatever you’re worried about, there’s no shortage of diets or foods purported to help you. Linking dietary habits and individual foods to health factors is easy — ridiculously so — as you’ll soon see from the little experiment we conducted.

But if you can’t spare the time to delve into Aschwanden’s 3,000-word description of that experiment, here’s the Cliff Notes listicle version of the problems she ran into. These all happen to be points we routinely raise in our reviews of news coverage of nutrition studies.

1. Questionnaire-based nutrition data are inaccurate

After trying to calculate her nutrition intake with the questionnaires used by researchers, Aschwanden found the logistics to be maddeningly difficult. For example:

Some questions — how often do you drink coffee? — were straightforward. Others confounded us. Take tomatoes. How often do I eat those in a six-month period? In September, when my garden is overflowing with them, I eat cherry tomatoes like a child devours candy. I might also eat two or three big purple Cherokees drizzled with balsamic and olive oil per day. But I can go November until July without eating a single fresh tomato. So how do I answer the question?”

2. “Positive” results are often false-positives

The problems with food questionnaires go even deeper. They aren’t just unreliable, they also produce huge data sets with many, many variables. The resulting cornucopia of possible variable combinations makes it easy to p-hack your way to sexy (and false) results, as we learned when we invited readers to take [a questionnaire] and answer a few other questions about themselves. We ended up with 54 complete responses and then looked for associations — much as researchers look for links between foods and dreaded diseases. It was silly easy to find them.

3. Correlation doesn’t equal causation

Our experiment found that people who trim the fat from their steaks were more likely to be atheists than those who ate the fat that god had provided for them. It’s possible that there’s a real correlation between cutting the fat from meat and being an atheist, Vieland said, but that doesn’t mean that it’s a causal one.

4. Benefits are overstated through reporting of relative risks

Relative risks are almost always much more extreme than absolute risk, but absolute risk (your risk of getting cancer if you consume bacon, for instance) is what we really care about. If, say, 1 out of 10,000 people who ate the most bacon got cancer, compared with 3 out of 10,000 who ate none, that’s a threefold difference. But the difference in absolute risk — a 0.01 percent chance of cancer versus 0.03 percent — is tiny and probably not enough to change anyone’s eating habits.

Why do we pay so much attention to research that has so much uncertainty baked into it?  Aschwanden speculates that it has something to do with a fundamental human desire for control: “The natural reaction when someone has a heart attack or is diagnosed with cancer is to look for a way to protect yourself from a similar fate. So we turn to food to regain a modicum of control. We can’t direct what’s going on inside our cells, but we can control what we put into our bodies.”

Aschwanden’s report suggests that in many cases, the control we seek through consumption of “good” foods and avoidance of “bad” foods is an illusion. Here at HealthNewsReview.org, we think that better reporting on study limitations can help the public avoid such magical thinking.


Kevin Lomangino is managing editor of HealthNewsReview.org and was formerly editor of Clinical Nutrition Insight. 

You might also like

Comments (2)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Craig Nelson

January 7, 2016 at 1:40 pm

Exactly right. And we’ve come to understand in the last decade or so that most of the dietary “truths” that we had come to rely upon were mostly bunk—for example the supposed benefits of a low-fat diet or of anti-oxidants in the diet.

And in the vacuum created by the fall of the low-fat diet others have rushed in—“No, it’s not low-fat that’s optimal, it’s low carb;”
“No, it’s Paleo;”
“No, it’s ketogenic,”
“No, it’s a plant-based diet;”
“No, it’s an organic, natural food diet;”
“No, it’s a raw-food diet.”

These all have in common with the low-fat diet that there is no good science supporting them. Rather than learning the correct lesson from the low-fat experience, these new diet gurus are simply repeating the same mistakes. What is curious is that no one has taken up the cause of what the science seems to be telling us:

Good health is compatible with a wide range of dietary habits.

If one avoids actual malnutrition (either from a calorie deficit, protein deficit or micronutrient deficit) you’ve done almost all that is to be done relative to your diet and health. We can find healthy, long-lived people all over the globe who eat very different diets for one another.

This strikes me as good news but somehow this is not the answer that people want. I suppose it’s attractive to imagine that one can controls one’s health destiny by fine-tuning your diet. But there is simply no evidence that such is the case.

Laurence Alter

January 12, 2016 at 3:28 am

Here’s a correlation you can consider (as a news writer): what per cent of cutsie-cheeky correlations do news writers employ in an effort of come across as clever or facetitious? (The two “charts” in your posted article, one about navels and cabbage). Taking an article seriously MEANS taking the entire article seriously sans childish humor.
Yours,
Laurence