March 11, 2015

Are P-values above .05 really just statistical noise? Part 2 of 3

Posted By

Tags

Dr. Gil Welch, Lisa Schwartz, P values, statistical significance, Steven Woloshin

When a journalist asked me a question about statistical significance recently, it opened my eyes to how little attention I’ve given the topic on this site. And as I started looking around, I found that there are some gems to guide understanding, but they’re not widely recognized.

Let’s look at a news example first.

9 years ago – I can’t believe it’s been that long – the Washington Post health section turned to Dartmouth’s Lisa Schwartz, Steve Woloshin and Gil Welch for a Healthy Skepticism column, “Fat or Fiction? Is There a Link Between Dietary Fat and Cancer Risk? Why Two Big Studies Reached Different Conclusions.”

It reflected on an “apparent flip-flop” in recent news about low-fat diet and breast cancer. One month, a front page Post headline read, “Low-Fat Diet’s Benefit Rejected: Study Finds No Drop in Risk for Disease.” But less than a year before, a headline sent a different message: “Study of Breast Cancer Patients Finds Benefit in Low-Fat Diet.”

The article addressed many concepts I can’t do justice to in this short blog post. And no need to, since you can read it yourself at the link above. But I draw your attention to this excerpt:

“Based on the size of the study groups and the number of cancers in each, the p value communicates how often you would expect to see an effect this big simply as a result of chance. By convention, scientists say p values below 5 percent are “statistically significant”— meaning not likely attributable to chance. And p values of 5 percent and higher are considered statistical noise (that is, likely due to chance).

The p values for the effect of low-fat diet on breast cancer in the two studies were quite similar. For women with breast cancer, the p value was 3 percent. For women without breast cancer, the p value was 7 percent.

So even though, by convention, one finding is called “statistically significant” and the other “not-significant,” we would say that the statistics of the two studies are not that different: Both are close to the conventional cutoff point of 5 percent. Since the p values are actually quite close, we would argue that the role of chance was about the same. That is, if you believe one is real, you should probably believe the other is real.”

Read the “Research Basics: Accounting for Chance” sidebar for an explanation of how close the two can be.

In part 3 of this series, learn from top biostatistician Dr. Donald Berry of MD Anderson Cancer Center, who wrote:

“Much of the world acts as though statistical significance implies truth, which is not even approximately correct.”

You’ll want to see what else he has to say.

————————

https://twitter.com/garyschwitzer

https://twitter.com/healthnewsrevu

and on Facebook.

Comments (3)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Richard Hoffman

March 16, 2015 at 9:10 am

While the p-value of less than 0.05 has been enshrined as the determinant of statistical significance, it is an arbitrary convention. In this case, the findings were both in the same direction so the results are not really different. One approach for reconciling apparently contradictory results is to perform a meta-analysis. Pooling results can increase sample size to provide more power to detect a significant difference. However, the study results are striking because the absolute magnitude of risk reduction was quite different—far greater in WINS than WHI. The reason is that these studies were asking different questions—WINS is evaluating the effect of low-fat diets for women with breast cancer while the WHI was conducted in women without breast cancer.

paul alper

March 16, 2015 at 10:20 am

I am an immense fan of healthnewsreview.org and often refer my fellow statisticians to the blog. However, the various attempts at defining p-value in this three-part presentation fall short. Here is the succinct definition:p-value is the probability of obtaining a result AT LEAST THIS EXTREME if the null hypothesis is true.

Prob(A AT LEAST THIS EXTREME has occurred | Null hypothesis is true)

The main (Bayesian) critique is that the wrong question is being asked. The correct question to ask is:
Prob(Null hypothesis is true | A has occurred)
Note that the correct question refers to A and does not involve more extreme results. Not too that p-value is, in effect, the wrong way round
Statisticians have fought over this for decades.
It should also be mentioned that the term “statistical significance,” which is completely synonymous with a low p-value, is linguistically misleading. The lay public tends to view it as even greater than significance without an impressive adjective–think of PENultimate being higher than ultimate. Much more important and valuable in the health field is “practical” or “clinical” significance. Even an exceedingly low p-value does not necessarily imply any practical or clinical change of heart.

David K. Cundiff, MD

March 16, 2015 at 11:24 am

It may well be that the amount of total fat in the diet and percentage of calories consisting of fat are of little or no significance in the incidence of breast cancer. However, the types of fats (e.g., animal versus plant-based or polyunsaturated verse saturated) may make a major difference. Since roughly two-thirds of fat in the U.S. diet is animal fat, the amount of total fat will correlate fairly strongly with animal fat and with saturated fat. However, it may well be that nuts, seeds, avocados, and olives protect against breast cancer while beef and dairy products strongly increase the risk. Observational studies need to be collecting the right data on the relationship of diet to breast cancer.

COMMENTS button We welcome comments, which users can leave at the end of any of our systematic story reviews or at the end of any of our blog posts.

But before leaving a comment, please review these notes about our policy.

You are responsible for any comments you leave on this site.

This site is primarily a forum for discussion about the quality (or lack thereof) in journalism or other media messages (advertising, marketing, public relations, medical journals, etc.) It is not intended to be a forum for definitive discussions about medicine or science.

We will delete comments that include personal attacks, unfounded allegations, unverified claims, product pitches, profanity or any from anyone who does not list a full name and a functioning email address. We will also end any thread of repetitive comments. We don”t give medical advice so we won”t respond to questions asking for it.

We don”t have sufficient staffing to contact each commenter who left such a message. If you have a question about why your comment was edited or removed, you can email us at feedback@healthnewsreview.org.

There has been a recent burst of attention to troubles with many comments left on science and science news/communication websites. Read “Online science comments: trolls, trash and treasure.”

The authors of the Retraction Watch comments policy urge commenters:

“Shed light, not just heat. Facts, challenges, disagreements, corrections — those are all fine. Attacking the person, instead of the idea or the interpretation, is neither acceptable nor helpful.”

We”re also concerned about anonymous comments. We ask that all commenters leave their full name and provide an actual email address in case we feel we need to contact them. We may delete any comment left by someone who does not leave their name and a legitimate email address.

And, as noted, product pitches of any sort – pushing treatments, tests, products, procedures, physicians, medical centers, books, websites – are likely to be deleted. We don”t accept advertising on this site and are not going to give it away free.

The ability to leave comments expires after a certain period of time. So you may find that you’re unable to leave a comment on an article that is more than a few months old.