Are P-values above .05 really just statistical noise? Part 2 of 3

When a journalist asked me a question about statistical significance recently, it opened my eyes to how little attention I’ve given the topic on this site.  And as I started looking around, I found that there are some gems to guide understanding, but they’re not widely recognized.

Let’s look at a news example first.

9 years ago – I can’t believe it’s been that long – the Washington Post health section turned to Dartmouth’s Lisa Schwartz, Steve Woloshin and Gil Welch for a Healthy Skepticism column, “Fat or Fiction?  Is There a Link Between Dietary Fat and Cancer Risk?  Why Two Big Studies Reached Different Conclusions.

It reflected on an “apparent flip-flop” in recent news about low-fat diet and breast cancer. One month, a front page Post headline read, “Low-Fat Diet’s Benefit Rejected: Study Finds No Drop in Risk for Disease.” But less than a year before, a headline sent a different message: “Study of Breast Cancer Patients Finds Benefit in Low-Fat Diet.”

The article addressed many concepts I can’t do justice to in this short blog post.  And no need to, since you can read it yourself at the link above.  But I draw your attention to this excerpt:

“Based on the size of the study groups and the number of cancers in each, the p value communicates how often you would expect to see an effect this big simply as a result of chance. By convention, scientists say p values below 5 percent are “statistically significant”— meaning not likely attributable to chance. And p values of 5 percent and higher are considered statistical noise (that is, likely due to chance).

The p values for the effect of low-fat diet on breast cancer in the two studies were quite similar. For women with breast cancer, the p value was 3 percent. For women without breast cancer, the p value was 7 percent.

So even though, by convention, one finding is called “statistically significant” and the other “not-significant,” we would say that the statistics of the two studies are not that different: Both are close to the conventional cutoff point of 5 percent. Since the p values are actually quite close, we would argue that the role of chance was about the same. That is, if you believe one is real, you should probably believe the other is real.”

Read the “Research Basics: Accounting for Chance” sidebar for an explanation of how close the two can be.

In part 3 of this series, learn from top biostatistician Dr. Donald Berry of MD Anderson Cancer Center, who wrote:

“Much of the world acts as though statistical significance implies truth, which is not even approximately correct.”

You’ll want to see what else he has to say.


Follow us on Twitter:

and on Facebook.

You might also like

Comments (3)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Richard Hoffman

March 16, 2015 at 9:10 am

While the p-value of less than 0.05 has been enshrined as the determinant of statistical significance, it is an arbitrary convention. In this case, the findings were both in the same direction so the results are not really different. One approach for reconciling apparently contradictory results is to perform a meta-analysis. Pooling results can increase sample size to provide more power to detect a significant difference. However, the study results are striking because the absolute magnitude of risk reduction was quite different—far greater in WINS than WHI. The reason is that these studies were asking different questions—WINS is evaluating the effect of low-fat diets for women with breast cancer while the WHI was conducted in women without breast cancer.

paul alper

March 16, 2015 at 10:20 am

I am an immense fan of and often refer my fellow statisticians to the blog. However, the various attempts at defining p-value in this three-part presentation fall short. Here is the succinct definition:p-value is the probability of obtaining a result AT LEAST THIS EXTREME if the null hypothesis is true.

Prob(A AT LEAST THIS EXTREME has occurred | Null hypothesis is true)

The main (Bayesian) critique is that the wrong question is being asked. The correct question to ask is:
Prob(Null hypothesis is true | A has occurred)
Note that the correct question refers to A and does not involve more extreme results. Not too that p-value is, in effect, the wrong way round
Statisticians have fought over this for decades.
It should also be mentioned that the term “statistical significance,” which is completely synonymous with a low p-value, is linguistically misleading. The lay public tends to view it as even greater than significance without an impressive adjective–think of PENultimate being higher than ultimate. Much more important and valuable in the health field is “practical” or “clinical” significance. Even an exceedingly low p-value does not necessarily imply any practical or clinical change of heart.

David K. Cundiff, MD

March 16, 2015 at 11:24 am

It may well be that the amount of total fat in the diet and percentage of calories consisting of fat are of little or no significance in the incidence of breast cancer. However, the types of fats (e.g., animal versus plant-based or polyunsaturated verse saturated) may make a major difference. Since roughly two-thirds of fat in the U.S. diet is animal fat, the amount of total fat will correlate fairly strongly with animal fat and with saturated fat. However, it may well be that nuts, seeds, avocados, and olives protect against breast cancer while beef and dairy products strongly increase the risk. Observational studies need to be collecting the right data on the relationship of diet to breast cancer.