The Cochrane Collaboration might be “Medicine’s Best Kept Secret” (but it shouldn’t be for journalists)


Just last month, after nearly three years of investigation, I published a book on a 22-year-old organization called the Cochrane Collaboration. The book’s subtitle, “Medicine’s Best Kept Secret,” was meant to be a little playful – as if one of the world’s largest organizations putting together systematic reviews of evidence around health care interventions could be a “secret.”

There are thousands of medical academics, researchers, clinicians and health policymakers who use Cochrane reviews every day in their work. The organization is also no secret to the 40,000 or so volunteers in over 130 countries who work under the banner of Cochrane, sifting, analyzing, and systematizing the body of evidence that underpins nearly every decision we make in healthcare. But alas, for the vast majority of patients and consumers of healthcare, the Cochrane Collaboration is a secret. I decided to write this book because I wanted to let the world in on the secret, which is vital both for the creators of health journalism as well as the consumers of health journalism.

The fact the Collaboration has become one of the world’s premier sources of independent and quality information about health care rests on a key principle: Individual studies to determine how well a health treatment works are not sufficient to make the best possible decisions. Higher up the evidence pyramid, you’ll find what the Collaboration produces: systematic reviews and meta-analyses which encompass all of the studies both published and unpublished on a particular question, studies that have been analyzed and statistically combined to create a summary of what is reliably reliable. When this is done in a methodologically sound and defensible way, what one hopes for is a consistent picture that points in the direction of benefit or harm related to that question.

I was approached by the organization in 2012 just prior to the 20th anniversary of the Collaboration and I wasn’t a stranger to its methods. I was periodically attending the annual meeting known as the Colloquium for at least a decade, and I was able to see — up close and personal — how these people operate. I was frankly inspired by a spirit of inquisitiveness and generosity that energizes their work. It was amazing to watch people who genuinely struggled with trying to find truth in health research, a process that’s messy, complicated, usually political, and always contentious, and motivated by an intense desire to turn large amounts of data into pictures that makes sense to doctors and to patients.

A recent example of the benefits of systematic reviews of evidence came across my plate when I saw the SPRINT (Systolic Blood Pressure Intervention) Trial. The study measured intensive blood pressure-lowering targets against “standard” blood pressure-lowering targets in people who had moderate hypertension. The trial was halted in September of this year and received a huge amount of media attention with the message that “lower is better.”  In other words, those patients who are given the more intensive blood pressure-lowering treatment did better than those who were given the standard treatment. For me the story doesn’t end there, because I asked the next question: What does Cochrane say?

For me, a single study is a chapter in a story that has a past, a history, and a trajectory. A new study is rarely going to be “game changing,” because you need to know how the new evidence fits into the existing systematized body of evidence — and that’s what Cochranites do.

The Cochrane Hypertension Review group at the University of British Columbia in Vancouver has looked closely at the ‘lower is better’ issue. They were able to take the results of this trial, add it to the 10 existing trials of lower vs. standard blood pressure targets, to conclude the big picture does not change.  Despite SPRINT’s breathless headlines around the world, the systematic evidence, which should be considered much more reliable, says that patients given the most intensive drug regimen to lower their blood pressure down to say 120 over 80 do not do any better than those who are given standard therapy. They don’t live longer or have fewer hospitalizations.  A review of new evidence in its proper context delivers huge rewards to the millions of people around the world–and their doctors–when they decide how aggressively they will pursue lowering their blood pressures.

[Editor’s note: The Cochrane Group’s conclusions regarding standard vs. lower blood pressure targets are expected to be published in January 2016. The results mentioned in the preceding paragraph are based on a preliminary analysis discussed in a telephone interview with Jim Wright, coordinating editor for the Cochrane Hypertension Review Group. These results should be considered tentative pending publication of the full updated Cochrane review.]

The “secret” in my book’s subtitle also refers to this idea: that even though the Cochrane Collaboration, outside of the world of academia or research, is largely unknown, it has, over the years, been able to expose some very troubling truths about what we think we know or don’t know about medicine and health care treatments.

One of my interviewees in the book explained it this way: Think of the body of medical procedures, screening programs, and drug treatments as a pie. If you were to divide that pie into thirds, the first third would contain all of those procedures or treatments that we know are underpinned by quality medical research and for which we can truly say with some degree of certainty that they “work.” The second piece of pie would contain those things we routinely do but we don’t have strong evidence that the benefits exceed the harms, because they haven’t been well studied. The last third would contain many things that we do in medicine and health care where there is evidence that they do more harm than good, and we should stop doing those things.

So whether you’re a patient or a journalist, you might want to ask yourself, “Which third am I looking at?” whenever you are presented with a new piece of research or an expert assertion about a treatment, a medical device, a screening tool, or a new medication.

As journalists we are there to provide context, and there is no better “context” for health care evidence than the body of 5,000 reviews in the Cochrane Library. The abstracts and the plain language summaries of all of Cochrane reviews are available on their website. The in-depth reviews are made available free to journalists who belong to the Association of Healthcare Journalists. They’re also available at most university libraries.

It’s worth taking a look at and it’s worth getting to know what is becoming, less and less, Medicine’s Best Kept Secret.

Alan Cassels is a pharmaceutical policy researcher, a member of the HealthNewsReview team, and author of the just-published book, The Cochrane Collaboration: Medicine’s Best Kept Secret (Agio, 2015).

You might also like

Comments (8)

We Welcome Comments. But please note: We will delete comments left by anyone who doesn’t leave an actual first and last name and an actual email address.

We will delete comments that include personal attacks, unfounded allegations, unverified facts, product pitches, or profanity. We will also end any thread of repetitive comments. Comments should primarily discuss the quality (or lack thereof) in journalism or other media messages about health and medicine. This is not intended to be a forum for definitive discussions about medicine or science. Nor is it a forum to share your personal story about a disease or treatment -- your comment must relate to media messages about health care. If your comment doesn't adhere to these policies, we won't post it. Questions? Please see more on our comments policy.

Jean-baptiste blanc

December 6, 2015 at 8:14 am

Cannot find the link to Cochrane were they report the systematic review including SPRINT. Can you give it ?

    Kevin Lomangino

    December 7, 2015 at 1:07 pm


    I have inserted the following editor’s note into the post in response to your question.

    Editor’s note: The Cochrane Group’s conclusions regarding standard vs. lower blood pressure targets are expected to be published in January 2016. The results mentioned in the preceding paragraph are based on a preliminary analysis discussed in a telephone interview with Jim Wright, coordinating editor for the Cochrane Hypertension Review Group. These results should be considered tentative pending publication of the full updated Cochrane review.

    Thanks for reading and commenting.

    Kevin Lomangino
    Managing Editor


December 6, 2015 at 1:13 pm

This is a very important issue, but I can’t find it – ‘the systematic evidence’ link gives only an old (2008-2009) publication

Laurence Alter

December 7, 2015 at 7:30 am

Along with PLoS One, it’s upper tier.

David K. Cundiff

December 7, 2015 at 12:28 pm

If the Cochrane Collaboration is the best kept secret in healthcare, then the systematic review on anticoagulants to treat venous thromboembolism that I led for Cochrane is the best kept secret in the Cochrane Database of Systematic Reviews: The 2 randomized trials in the review were both negative (more deaths in anticoagulated patients than in non-anticoagulated patients. Yet we concluded, “Since the use of anticoagulants is widely accepted in clinical practice, designing and implementing other similar studies would not be ethical.” My opinion piece, “Evidence-based Medicine and the Cochrane Collaboration on Trial” was ignored by the medical media: Also ignored was my “Systematic review of Cochrane anticoagulation reviews”: My “Citizen Petition” to the FDA to retract FDA approval for anticoagulants for venous thromboembolism has been under review for the past 5 months:!documentDetail;D=FDA-2015-P-2373-0001 No anticoagulation experts, including those that write the guidelines for anticoagulation treatment for venous thromboembolism will rebut my contention that anticoagulants don’t benefit patients with venous thromboembolism. Since August 2015, the Cochrane Oversight Committee, Chaired by Dr. Richard Smith, former Chief Editor of the BMJ, has been considering my appeal for them to retract a Cochrane review of non-inferiority trials of deep venous thrombosis treatment comparing “standard anticoagulants” with novel anticoagulants. I called for the retraction because “standard anticoagulants” are not evidence-based to work for deep venous thrombosis. Dr. Smith promised me that his Oversight Committee would make a “preliminary” statement about my appeal by Christmas. My call to the BMJ Chief Editor, Dr. Fiona Godley, to retract a systematic review of non-inferiority trials regarding venous thromboembolim comparing standard anticoagulants with novel anticoagulants:
A healthcare reporter is needed to ask some hard questions of the Cochrane hierarchy.

    Beverly Richards-Smith

    December 8, 2015 at 9:43 am

    @David K Cundiff: “The use of anticoagulants is widely accepted in clinical practice*, so a further RCT comparing anticoagulants to placebo could not ethically be carried out.” *The same could have been said of bloodletting, trepanning, prefrontal lobotomy and a variety of other ineffective and/or harmful medical practices that are now regarded with horror. I’m glad to know someone is willing to challenge the Cochrane Group on this issue. I cannot access their analyses of studies on HMG CoA inhibitor (statin) use in the prevention of cardiovascular events or stroke, which I suspect may fall into the same “unchallengeable because it’s standard practice” chasm. What happened to the concept of “evidence-based medicine?”

Richard Morley

December 8, 2015 at 10:53 am

Thank you for this interesting and insightful article. I am the Cochrane Consumer Network Coordinator. For healthcare consumers and others looking for evidence a good place to start is where you can find plain language summaries of Cochrane reviews. Readers might be interested to know that there is a network of consumers (patients, carers and family members) who contribute to producing Cochrane evidence. More information about the network and how to join can be found here

Harri Hemila

December 11, 2015 at 9:27 am

HealthNewsReview tends to be critical, but the description of Cochrane Collaboration is not so.
I will give three brief examples.

1. Cochrane Database published a review “zinc for the common cold” in 2011. I wrote a feedback in which I listed 10 errors in that review. The review was updated in 2013, but only one of the errors was corrected. Thus, the Cochrane editors did not check whether the errors that I pointed out had actually been corrected. Furthermore, there also seemed to be unattributed copying of text and data in the 2013 version, so I wrote a document in which I described the errors of the 2013 version in detail:
Thereafter the review was withdrawn but there are no descriptions about the reasons for withdrawal in PubMed or in the Cochrane Database
That Cochrane review substantially misled its readers.

2. Cochrane Database published a review “vitamin C supplementation for asthma” in 2001. I read the 2009 update of that review and found out that there were severe errors in extraction of data and in data analysis. I wrote a feedback in which I calculated that there was strong evidence from three RCTs on exercise-induced bronchoconstriction [EIB] that vitamin C was better than placebo with P = 0.00007 whereas the Cochrane authors had concluded that there was no benefit of vitamin C. A response to my feedback was published in 2012, ie with a long 3 year delay.
The response did not properly respond to my criticism and had errors of its own. I wrote a new feedback, but Cochrane editor rejected my second feedback with comment “It is not the policy … for the feedback section of Cochrane Reviews to serve as a forum for commentators to debate whether they agree with the way in which the authors of the review have responded to their previous comment.”
Such argument is inconsistent with Cochrane’s own principles and COPE guidelines.
Cochrane Collaboration’s own principle #8 states:
“… being open and responsive to criticism.”
COPE Codes of Conduct states that
1.1. Editors should be accountable for everything published in their journals
1.8. [editors should] always be willing to publish corrections, clarifications, retractions and apologies when needed
See my two feedbacks in:
Based on my first feedback, I wrote two papers of the 3 RCTs on vitamin C and EIB and there has not appeared any letters-to-editor suggesting that my statistics in those papers might be unsound
The first version of the Cochrane review vitamin C for asthma was published in 2001. Later I found that that first version already had the errors that I pointed out in 2009. Thus that Cochrane review misled its readers for a decade.

3. Cochrane Database published a review “Antioxidant supplements for prevention of mortality in healthy participants and patients with various diseases” in 2008. I wrote a feedback in which I pointed out that there was a serious “apples and oranges problem” in that review.
“Antioxidant” is a wide category and pooling all diverse antioxidants together makes as little sense a pooling all “antibiotics”. We should not ask “what is the effect of antioxidants” but we should ask “what is the effect of vitamin E, and the effect of vitamin C, etc”. However, even that question is not appropriate, since there is strong evidence that the effects of vitamin E are not uniform, but vary between population groups. I pointed out this problem in 2007 in a letter to JAMA as a comment to the earlier version of the Cochrane review that was published in JAMA.
Thus, when the authors wrote the 2008 Cochrane review, they were aware of the strong evidence that the effects of vitamin E were heterogeneous yet they ignored that issue. See the problems in that Cochrane review in:
In 2009 I showed that there was strong evidence that the effect of vitamin E on total mortality was heterogeneous in the ATBC Study. It increased mortality in young participants but decreased in older participants. Thus, the effect of vitamin E on mortality is not uniform and cannot be described with a single estimate of effect:
Thus, pooling all diverse antioxidants together and calculating a single estimate of “antioxidant effect” misleads readers.

There are many more Cochrane reviews that I have found to be poor and misleading but I cannot describe more than these three which are close to my own research topics.
I am a Cochrane review author myself and my review vitamin C and the common cold has been among the most read Cochrane reviews over a decade.
There are also many good Cochrane reviews. Two particular benefits of the unconventional publication format with regular updates that errors can be corrected, and new data and other new information can be included.
However, many Cochrane authors have poor understanding of basic statistics and some are also careless. Many Cochrane authors are enthusiastic as Alan Cassels describes, but that does make a person scientifically or statistically competent to write good systematic reviews.
The validity of Cochrane reviews should be considered case by case and readers should not consider that all of them are as good as Cassels assumes.