January 5, 2017 Read Original Release

An app that delivers depression care “as fast as finding good sushi”? Northwestern should know better

Reviewed By

Rating

New apps designed to reduce depression and anxiety as easily as checking your phone

Our Review Summary

This release on a suite of smartphone apps designed to reduce depression and anxiety comes from a prestigious university and was funded by the National Institutes of Health. Issued under embargo, an emailed version of the release pitched journalists with the titillating subject heading: “Depression Apps as Fast as Finding Good Sushi.”

But if Northwestern University wants us to think of this app as the Yelp of depression treatment, it will need to serve up more convincing evidence than what was provided in this skimpy news release. After all, treating depression is a little more complicated than locating a good California roll, and we expect to see substantial backing for the suggestion that these apps work as well as clinician-delivered psychotherapy or antidepressant medication. The release’s comparison of the app to medication is fatally flawed because more than half of the participants were already taking antidepressants when they enrolled in the study, and about 20% were receiving psychotherapy. How much of the benefit was due to the app and how much to these or other factors is impossible to say.

Bottom line: while the release deserves some credit for highlighting one major weakness of the study — the lack of a control group — it didn’t mention other key limitations or adequately describe the scope of the benefits seen. Like the sushi lovers courted in the subject heading, anyone feasting on this release is likely to be hungry again in an hour — if not sooner.

Why This Matters

People are being pitched on new health-related smartphone apps on a weekly basis. By now, almost everyone is aware that that they can access health interventions via the Internet and one would hope these apps are becoming more evidence-driven. Efforts such as Northwestern’s to give more people access to coping tools while assessing their effectiveness is laudable and forward-looking.

It’s even more impressive when considering that there’s scant evidence to back up claims made by scores of developers of apps for people who need help dealing with anxiety and depression. According to an April 2016 feature article in Nature, “But the bare fact is that most apps haven’t been tested at all. A 2013 review identified more than 1,500 depression-related apps in commercial app stores but just 32 published research papers on the subject. In another study published that year, Australian researchers applied even more stringent criteria, searching the scientific literature for papers that assessed how commercially available apps affected mental-health symptoms or disorders. They found eight papers on five different apps.”

Criteria

Does the news release adequately discuss the costs of the intervention?

Satisfactory

The release notes that there is no cost associated with downloading the Intellicare apps. But because Intellicare is part of “a national research study” the researchers request that users provide “confidential feedback” weekly to help them further develop the system. Some people may not be comfortable sharing personal data over the internet, with good reason.

Does the news release adequately quantify the benefits of the treatment/test/product/procedure?

Not Satisfactory

The release makes several claims but doesn’t back them up with data. For example:

“13 novel mini-apps reduced depression and anxiety by nearly 50 percent in a preliminary research study”

What kind of depression and anxiety are we talking about here? Feeling a little blue or so severely depressed you can’t get out of bed? Which tools were used to measure or rate participants’ level of depression or anxiety?

“participants reporting significantly less depression and anxiety by using the apps on their smartphones up to four times a day”

Again, what does “significantly” refer to?

“The short-term study-related reductions are comparable to results expected in clinical practice using psychotherapy or with that seen using antidepressant medication.”

Claiming that using the apps for 8 weeks is comparable to a similar period of psychotherapy or antidepressant use is unfounded without a randomized clinical trial. The study itself notes that 64% of participants were taking antidepressant medication, and 22% were receiving psychotherapy, when they enrolled in the study.

Does the news release adequately explain/quantify the harms of the intervention?

Not Satisfactory

Although there’s little harm associated with using any app on the face of it, relying on an app instead of seeking professional help could be harmful. The release should have acknowledged this, and done a better job of explaining who is most likely to be helped by a smartphone app and which individuals have symptoms severe enough to warrant seeking professional medical assistance. There are many different types of depression that range in severity. The National Institute of Mental Health describes several — seasonal affective disorder, persistent depressive disorder, major depressive disorder, psychotic depression, postpartum depression.

Does the news release seem to grasp the quality of the evidence?

Satisfactory

The release notes that 96 of the 105 initial volunteers completed the study and that participants received coaching (including phone calls and texts) during the 8 week study. But it didn’t explain that maybe it was this human coaching, and not the apps per se, that was at least partly responsible for the benefits seen in the study. The researchers acknowledge in their manuscript that “we cannot disentangle the effects of the coaching and the apps.”

The release also could have done a better job of highlighting the limitations of such a short study. Will users get bored with the app and see their depression/anxiety return when they stop using it?

We’ll give credit to the release, however, for nicely describing a major weakness of the study. It states: “The preliminary study did not include a control arm, so it’s possible that some people who enrolled in the trial would have improved anyway, partly because they may have been motivated to try something new, Mohr said.”

Does the news release commit disease-mongering?

Satisfactory

The release doesn’t go overboard describing the prevalence of depression in our society. It reasonably states, “More than 20 percent of Americans have significant symptoms of depression or anxiety each year, but only around 20 percent of people with a mental health problem get adequate treatment.”

Does the news release identify funding sources & disclose conflicts of interest?

Satisfactory

The release notes that the study was funded by the National Institute of Mental Health.

Does the news release compare the new approach with existing alternatives?

Not Satisfactory

The release states that over the course of 8 weeks, participants experienced improvements similar to what people with depression or anxiety would find through psychotherapy or medication. But again,how can such a comparison be made when 64% of participants were already taking antidepressant medication, and 22% were receiving psychotherapy, when they enrolled in the study?

Does the news release establish the availability of the treatment/test/product/procedure?

Satisfactory

The release invites people to download the apps for free.

Does the news release establish the true novelty of the approach?

Not Satisfactory

The releases says, “Most apps designed for mental health typically offer a single strategy to feel better or provide too many features that make them difficult to navigate. Users may get bored or overwhelmed and may stop using the apps after a few weeks.”

This statement doesn’t adequately explain why the same problems wouldn’t apply to the apps described in the release. (After all, the study was only 8 weeks long.) Nor are we told how these apps differ from the thousands of others on the market.

Does the news release include unjustifiable, sensational language, including in the quotes of researchers?

Satisfactory

The release doesn’t rely on sensational language, although the sushi analogy was ill-advised.

Total Score: 6 of 10 Satisfactory

Comments

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

COMMENTS button We welcome comments, which users can leave at the end of any of our systematic story reviews or at the end of any of our blog posts.

But before leaving a comment, please review these notes about our policy.

You are responsible for any comments you leave on this site.

This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.

We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.

We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.

There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”

The authors of the Retraction Watch comments policy urge commenters:

“Shed light, not just heat. Facts, challenges, disagreements, corrections — those are all fine. Attacking the person, instead of the idea or the interpretation, is neither acceptable nor helpful.”

We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.

And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.

The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.