Poking holes in Pokémon Go health benefit claims–including a study that didn’t exist

We’ve been tracking some wild and unsubstantiated claims of health benefits from the use of the Pokémon Go app. A quick web search turns up thousands of articles with headlines that read more like advertising than journalism – making claims about impact on obesity, type 2 diabetes, anxiety, depression, and more.

Stories have:

• called it “a health and wellness app.”
• claimed that its “mental health benefits are real” and that “The research is really, really clear on this.” (Those statements refer to exercise, not to the app game. Only long-term evidence will show how many people will stick with the app for time spans long enough to gain a benefit.)
• cited anecdotes of people claiming that “for those with depression, who can have difficulty finding the energy to leave the house, the new game has been a motivating force. People with social anxiety are meeting other players in the context of play, where social interaction can seem less daunting.”
• claimed that it “is having an amazingly positive side effect on players’ fitness and mental health.”
• claimed that it “is unexpectedly great for mental health.”
• claimed that it “has everyone exercising” Everyone?  I don’t know a single person in my family or circle of influence that is using the app.

Many of these stories relied on anecdotes, often single anecdotes, often pulled from social media.

A university news release made the unsubstantiated claim that “Real-life positive health consequences of playing Pokémon Go are happening across the nation.”

Uh-oh: A study that doesn’t actually exist

But the icing on the cake of hype is a story posted by Nature World News headlined, “‘Pokémon Go’ Could Help Prevent Type 2 Diabetes, Study Finds.”  In the text of the story, it states: “According to the study, ‘Pokémon Go’ users are leaving their homes to walk for miles by just playing the game, engaging in intense physical activity without them noticing.”

Fascinating.

Especially since there was no study.

The story was based on a University of Leicester news release that never mentioned a new study.  It did include the sweeping claim that “Pokémon Go could ease Type 2 diabetes burden.” But the basis for the release was not a study, but mere speculation by a researcher in response to people going ape over the app. The researcher was quoted: “If there is something out there which is getting people off the sofa and pounding the streets then this game could be an innovative solution for rising obesity levels.”

Yes – “if there is something out there” it “could be an innovative solution.” IF and COULD BE are the operative words.  Nowhere did the news release state that this was based on a study. Because there was none.

We’ve written in the past about claims about studies that didn’t exist – once last month and once last year and once four years ago.

‘Real…amazing…unexpectedly great’

Consumers and patients must learn to be skeptical and analytical about any claim anytime from anyone that begins, “A new study finds…..”

First, ask yourself if there was even a study. The writer may neglect to tell you that, and, worse, may even mislead you into thinking there was one.

It is clear that anecdotes are piling up about some people using the app to get out and exercise more.

But let’s not overstate the evidence about benefits that are “real…amazing…unexpectedly great…(for) everyone.”  And journalists, please remember that it’s your responsibility to independently vet claims about new ideas – not to serve as stenographer for any researcher or any university PR person who finds a way to get publicity by weighing in on a fad.

As an example of how this can be done, read a Vox story that raised caveats and reminders about what’s not known about the use of the app–“an unlikely fix for the obesity epidemic…surge in use may be temporary…reviews of evidence on video games designed to get people active and improve health reveal a mixed picture of their effectiveness.”

4 experts’ comments

And here’s what journalists might have found if they simply looked for independent analysts:

• Dr. Michael Joyner of the Mayo Clinic, whose research often focuses on exercise physiology, wrote me: “the crux of the matter with anything or any approach to get people to be more active, change their diet or do other positive lifestyle things is long term adherence.  So if Pokemon Go gets people to be more active and if they stay more active over years – not days or months – then there likely will be positive health effects.  Beyond that it is simply a lot over extrapolated hype.  Change the words “Pokemon go” to “ a gym membership”, “buying exercise equipment”, “a wearable device”…… you name it and the story is the same.  Over and over and over again and again.”
• Timothy Caulfield, Canada Research Chair in Health Law & Policy, University of Alberta:  “Anything that helps people get active is terrific, but is there good evidence of positive impact of Pokemon Go? These claims assume that walking around will result in clinically significant weight loss.  We know well that the relationship between physical activity and weight loss is complex. (In general, physical activity alone is not an effective strategy.)  Are they eating less and better because of Pokemon Go?  Also, is this sustainable? How long will Pokemon Go help people remain active?”
• Psychiatrist Susan Molchan:  “A study to show health benefits—at least in the form of a bit of weight loss or better blood glucose or blood pressure control would need to be randomized between groups that played the game and those who played a sedentary game. Even then results would be suspect,as such a study, to show real health benefits, should go on for at least a couple of years, and of course it is played primarily in young people, who do not as a rule have health problems, so it would be even harder to show a “benefit” to a control group compared to a less healthy,older control group.
  Interestingly, I became familiar with the game in the last week or so as my daughter started playing on vacation. You don’t necessarily have to be active to capture Pokemons, one of our most productive sites was at a red light in a car.”
• Family medicine & weight management specialist Dr. Yoni Freedhoff: “Since many of the stories were about weight loss, I can’t help but wonder how many extra calories were consumed by people who believed, because they strolled around for a few hours hunting for Pokemon, that they needed, or earned, a treat or two.”

Hat tip to Ivan Oransky of MedPage Today for pointing us toward http://www.obesityandenergetics.org – and to that website, for capturing the “what is it really a study?” episode.