This flu season, let’s immunize ourselves from the annual infection of exaggerating relative risk reductions  

The following post is by Alan Cassels, a regular contributor to the blog who is also a drug policy researcher at the University of Victoria, Canada.


vaccine 410 x 410“Normally, the flu vaccine is between 50 to 60 percent effective”

— Dr. Tom Frieden, Director of the Centers for Disease Control and Prevention (CDC)

How effective is the flu shot?   

That’s an important question that generates many headlines across North America every fall as the public health community starts ramping up its fall flu campaigns. Problem is, the media tends to generate a lot of noise around that number, but provide very little clarity.

Influenza stories this time of year swirl around similar themes, often with the CDC taking center stage (featuring the platitudinous photo op of the CDC director rolling up his sleeve for the shot), including the importance of the flu shot, the need for general hand hygiene and the expected effectiveness of the vaccine this year.

The number that arrives as predictably as the fall flu campaign is “60” as in this story that repeats the mantra: “Flu shots normally prevent 60% to 65% of infections serious enough for people to see a doctor.” Across North America, public health officers will stress that the flu vaccine “reduces the risk by 60%!” Sixty sounds good. Impressive. Powerful. But ultimately as meaningless as a used car salesman with big bright signs of “60% off” plastered on every car on the lot, without ever telling you what the regular prices are. Despite the potential for a huge bargain on that purchase, the reality is that you’re in for a much smaller deal than you think you are. MUCH smaller.

And so it goes with the flu shot. When people hear “60% reduction,” I believe this is what happens inside their thought bubble: “If my risk of getting the flu this winter is 100%, the shot will reduce that to 40%. So instead of 100 people getting the flu, only 40 would get it. Hmmm. This 60% reduction sounds like great odds.”   

Even when the flu vaccine seems less effective, like in this story which said that “last year the flu shot, by the CDC’s own numbers, was 23% effective,” people might think it’s a good deal. 23% off that Chevy Impala in the back lot might be a great bargain. But what does the 23% or 60% really mean?  

Since they are relative numbers, they demand us to ask “23% of what?” or “60% of what?”  As we’ve made abundantly clear at HealthNewsReview.org, using relative numbers on their own are a statistical no-no because, to quote ourselves, “we think the relative risk number tends to inflate the impression of how much impact the drug [or vaccine] has.” For a quick refresher on relative numbers check out our review criteria here.

I find the 60 or the 23 percents irritating, not to mention epically misleading and I’m not alone. One commentator looking at a flu study reporting a “24% risk reduction” called it “Cockamamie propaganda”. Colorful but true.  

To get some perspective I talked to Dr. Tom Jefferson in Rome who has reviewed hundreds of flu vaccine studies as part of the Cochrane Collaboration. Dr. Jefferson gives me a quick tutorial on the 60% which he calls “CDC / pharma spin of the worst kind.”

He reminded me that every flu season there are over 200 viruses which can cause influenza and influenza-like illness, all perfectly capable of making you headachy and feverish. Most people get through the flu just fine and, thankfully, the risk of death or serious illness in otherwise healthy people is rare. In a good year the vaccine might protect you against influenza A and B, which might make up about one tenth of all circulating viruses.

The best way to assess flu trials is to look at those that compared vaccinated people with unvaccinated people.  

When Jefferson and his colleagues published their March 2014 review they found that under ideal conditions (when the vaccine matches the main viruses circulating that season) you need to vaccinate 33 healthy adults to avoid one set of influenza symptoms. This is what we’d call a NNV (Numbers needed to Vaccinate) of 33. When the vaccine match isn’t so good as it was last year, the NNV is about 100. That is, of 100 people vaccinated, 99 will have no benefit and one person will avoid one set of influenza symptoms. Vaccination did not seem to affect the number of people hospitalised or who lost working days.     

Almost half (15 of the 36 trials they examined) were funded by vaccine companies and four had no funding declaration. His team cautioned that even these numbers may represent an “optimistic estimate” because “company-sponsored influenza vaccines trials tend to produce results favorable to their products.” You can read more details here.

As for the magical “60?” Dr. Tom Jefferson didn’t mince words: “Sorry I have no idea where the 60% comes from – it’s either pure propaganda or bandied about by people who do not understand epidemiology. In both cases they should not be making policy as they do not know what they are talking about,” he said, insisting that I quote him.

When asked to explain the 60% number via email, the CDC sent me a link to their website, which states that the 60% figure is a “point estimate” of laboratory-confirmed flu illness that results in a doctor’s visit or urgent care visit, presumably derived from studies such as this one showing a “61%” effectiveness rate for the 2013/2014 shot.

If you don’t go beyond the “60%” headlines, you probably wouldn’t question the vaccine because if A) the vaccine is so effective, B) the vaccine is ‘free’ or almost free; and C) it’s relatively safe, then how could you say no to that?

Doesn’t that equation change if the effectiveness is between 1 to 3 percent, depending on how well this season’s circulating virus has been matched with the new vaccines? 

As the CDC continues to stress that “a yearly flu vaccination is still the best protection currently available against the flu,” you can imagine the confusion playing out in the thought bubbles of the general population.  

What I long for—and I haven’t seen it yet—is for media coverage this season to start reporting on absolute differences related to the flu vaccine. I’d like to see how the “1-3% effectiveness of the vaccine” floats around in the public’s thought bubbles. How does that compare with something as simple as staying home and not infecting other people or washing your hands more frequently?

I think if the real numbers were out there, we might see a much broader public conversation about what other sorts of flu “protection” are worthwhile.

You might also like

Comments (10)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Robert Quickert

October 13, 2015 at 3:12 pm

«One commentator looking at a flu study reporting a “24% risk reduction” called it “Cockamamie propaganda”. Colorful but true.» The link given leads to a blog post by Dr. Brownstein, MD. the bio at teh bottom of the post says «He is the Medical Director of the Center for Holistic Medicine in West Bloomfield, MI. Dr. Brownstein has lectured internationally to physicians and others about his success in using natural hormones and nutritional therapies in his practice.» Such quackery does not inspire confidence as to the quality of Dr. Brownstein’s scientific or medical opinions, and hence nor those of Mr. Cassels.

    Rose

    October 20, 2015 at 4:05 pm

    Robert Quickert…he is also a Board Certified Family Physician. What is your background?
    Diet (nutritional therapies) are involved in heart disease, diabetes, (two biggies) weight loss, Phenylketinuria, Galactosemia, some epilepsies…I’m just a housewife so I can’t think. Melatonin has been used for years to help people sleep, thyroid hormones, growth hormones, insulin is a hormone, let’s see…Yep, he’s an idiot.

Robert Quickert

October 13, 2015 at 3:20 pm

Can Mr. Cassels clarify this statement: “Almost half (15 of the 36 trials they examined) were funded by vaccine companies and four had no funding declaration.” The link to the Cochrane Collaboration review he provides has in its Author’s conclusions section: “This review includes 90 studies, 24 of which (26.7%) were funded totally or partially by industry. Out of the 48 RCTs, 17 were industry-funded (35.4%).” Thanks.

justin

October 14, 2015 at 11:43 am

Since your article is about he mythical “60%”, you should explain what a ” “point estimate” of laboratory-confirmed flu illness that results in a doctor’s visit or urgent care visit” means.

    Redpill1

    October 15, 2015 at 8:17 am

    mythical “60%”, you should explain what a ” “point estimate” of laboratory-confirmed flu illness that results in a doctor’s visit or urgent care visit” means.”
    -Actually the CDC should explain what that means since it’s a number they pulled out of the air. The CDC pulls lots of number out of the air like the lie that many websites are repeating that 36,000 people in the US die from the flu and that last yer 1,000 children died from complications of the Flu. If you press the CDC on these numbers and ask them to clarify what “complications from the flu” means they send you a link to one of their webpage. The CDC and the government has spent many decades conditioning people to see and read only what they want them to. When you see a statement like “died from complication of the flu” the only thing parents see are died & flu so they rush out to get the kid vaccinated. People need to challenge the CDC and tell them to communicate in plain English.

Daniel Pendick

October 19, 2015 at 1:51 pm

Excellent breakdown on that 60% figure from CDC. I’ve always felt uncomfortable parroting that in my reporting. But I just want to point out that you did not link the “1 to 3 percent” statement clearly enough to the earlier reference to NNV. For a lot of readers, that will not be completely obvious. They will not make the connection. They will think, “One to three percent huh?”

Aleta Kerrick

October 19, 2015 at 8:09 pm

For us folks who aren’t epidemiologically informed, could you please add to the article an explanation of the CDC’s explanation of their 60% figure? What’s a “point estimate”? How is their explanation adequate/inadequate?

Liz Scherer

October 20, 2015 at 6:12 am

This is a fascinating piece. However, I wish that you had added a few sentences on context, i.e. apples/oranges with regard to vaccinations. When I read this, my take away was not only that the benefits of flu vaccinations were exaggerated but my mind also went directly to the association between the CDC and industry. While the association derives from a quote, it also indirectly fuels a larger fire, one between vaccines and autism. I know; it’s a stretch. However, context matters and it behooves the author to raise that point.

    Alan Cassels

    October 20, 2015 at 5:31 pm

    I’ll respond in the order these were received.
    Robert Quickert’s request that I clarify this statement “Almost half (15 of the 36 trials they examined) were funded by vaccine companies and four had no funding declaration.” And this statement: “This review includes 90 studies, 24 of which (26.7%) were funded totally or partially by industry. Out of the 48 RCTs, 17 were industry-funded (35.4%).” A Cochrane review includes observation studies and randomized trials, some worthy of inclusion in the review and others not. The subset of 36 trials examined (ie: included in the review) had 15 that were industry funded. Both statements are true. As to the suggestion that Dr. Brownstein practices quackery is unworthy of comment.
    Justin and Aleta Kerrick asked me to explain the “60%” point estimate and Redpill1 says it’s a number pulled from the air. I contacted CDC, searched their literature, as well as the major studies that I think would be the basis for the “60%” and frankly can’t confirm where it comes from. I know it’s a relative risk reduction averaged from a series of studies, but I don’t know which ones the CDC are using and they wouldn’t tell me. This link might explain some of the absolute numbers, but I have no way to know if the CDC’s 60% is based on these numbers http://blog.minitab.com/blog/adventures-in-statistics/how-effective-are-flu-shots

    I agree with Redpill1 that the numbers of so-called deaths due to complications of the flu can be misleading and be interpreted as fearmongering.

    As for Mr. Pendick, I thought the “1 to 3 percent” statement was a fairly clear connection to the Numbers Needed to Vaccinate (calculated as 1/ ARR, as in 1/.01 and 1/.03) and I hope they do make the connection that the benefits are between one and three percent and that between 33 and 100 people are needed to vaccinate to prevent one case of flu symptoms.
    As for Liz Scherer’s comment, about me adding context. I ‘d like to do that though I’m not sure what you mean. There may be many explanations why the benefits of flu shots are often presented in relative terms that tend to exaggerate their effectiveness but I am not sure the cosiness between the CDC and industry is the strongest one. I wonder if other readers have any theories.

Victoria Sweet MD PhD

October 22, 2015 at 12:13 pm

I looked this up once. Here is how they get to 60%: If 100 unvaccinated people are exposed to the influenza virus, 5 will get the flu. If all 100 are vaccinated and then exposed to the influenza virus, 3 will get the flu. So 3/5 is 60% effective. As usual no calculation is wasted on the side effects or adverse reactions. Not to speak of the costs, the hassle factor, or the possible long-term effects of not exercising your immune system, ever.