Observational Studies and Falsification Endpoints

Posted By


Last week’s Journal of the American Medical Association included an article, “Prespecified Falsification End Points: Can They Validate True Observational Associations?” that got guest blogger Harold DeMonaco, MS, thinking in a way that might get you thinking.  Here is his guest post:


That JAMA article by Prasad and Jena offers a rather unique solution to the vexing problem of false positive associations generated in observational studies.  Their solution is to include an implausible hypothesis into the mix, called a “falsification endpoint.”

The current New Drug Approval (NDA) process is woefully inadequate to identify relatively rare side effects of prescription drugs under review.  The reason is fairly simple.  There are too few subjects (patients) enrolled in the seminal trials.  If an adverse event has an incidence of 1 per 1000 patient years, there is simply no way that it would be picked up on review.  Unless of course, the side effect is sufficiently rare (e.g. Progressive Multifocal Leukoencephalopathy seen in patients treated with Tysarbi) as to jump up and grab the observer by the throat.

The increasing availability of large electronic medical databases provides a great opportunity for researchers to look for hypothesis generating observations and to identify potential rare side effects.  But size does not always provide greater clarity.  For example, several observational studies suggested an association between the use of acid reducing proton pump inhibitors and pneumonia.  As is noted by the JAMA article authors, there is a biologic underpinning.  De-acidification of the stomach allows what a colleague once described as “colon-ization” with bowel bacteria now residing in the neutralized stomach.  Aspiration due to reflux disease would seem to be a plausible etiology for the apparent increase in the incidence of pneumonia.  Subsequent studies however failed to demonstrate an increased risk.

Prasad and Jena offer an out of the box solution.  In addition to testing the desired association, test one or more that cannot possibly be true.  Introducing impossible associations into the statistical analysis reduces the likelihood of false positive results.

  • “For instance, a falsification hypothesis may be that PPI (proton pump inhibitor drug) use increases the rate of soft tissue infection or myocardial infarction. A confirmed falsification test—in this case, a positive association between PPI use and risks of these conditions—would suggest that an association between PPI use and pneumonia initially suspected to be causal is perhaps confounded by unobserved patient or physician characteristics.”

They also suggest that the coincident analysis of know side effects seen with a drug further validates study results.

  • “For instance, an observational study suggesting an unknown adverse effect of clopidogrel (for example, seizures) should also be able to demonstrate the presence of known adverse effects such as gastrointestinal hemorrhage associated with clopidogrel use. The inability of a study to verify known adverse effects should raise questions about selection in the study population.”

The coincident search for either plausible association (e.g., rash with an antibiotic, bleeding with an antiplatelet drug) or a truly implausible association (e.g., monocular blindness with penicillin) provides additional insight into the validity of the sought after association. Or, in layman’s terms, “If you want to look for an association between a drug and a side effect in a population, also look for a side effect that is totally unlikely or one that has been demonstrated previously.”  It will be interesting to see if this approach gains any traction in the medical literature.


Follow us on Facebook, and on Twitter:






You might also like


Please note, comments are no longer published through this website. All previously made comments are still archived and available for viewing through select posts.

David egilman

January 21, 2013 at 4:54 pm

Excuse me propulsid an antacid caused cardiac deaths

Harold DeMonaco

January 22, 2013 at 2:28 pm

I would agree at least in part. Propulsid (AKA cisapride) is a gastroprokinetic agent that increased tone of the gastroesophageal junction (thus reducing the risk of acid reflux) and increased gastric motility. It is not a PPI nor is it an antacid. It was withdrawn from the market when it was recognized that there was an increased risk of cardiac arrhythmias (Torsades de Pointes) and death especially when used in combination with drugs that altered a metabolizing enzyme (cytochrome P450 3a4) such as erythromycin, clarithromycin, certain antifungal drugs and anti-HIV drugs.