Failure to Replicate: A Big Nail in the Animal Research Coffin

Diabetes researchers sat up straight in 2017 when two mouse studies reported the ability to increase pancreatic insulin production with drugs: the neurotransmitter GABA and the anti-malaria drug artemisinin that works through the GABA pathway. Appearing in the prestigious journal Cell, these reports caused excitement that a means for reversing type 1 diabetes mellitus was at hand.

But hold the headlines—neither outcome could be replicated in carefully designed studies at the University of Pennsylvania Perelman School of Medicine and the University of California Davis, both published in another prestigious journal, Cell Metabolism. These reports led pancreatic biology researcher Fred Levine to state: "These two recent papers in Cell Metabolism have shown definitively that the two earlier papers in Cell are completely irreproducible."

This failure to replicate is not unusual. The irreproducibility of animal research seems to be present wherever it is investigated, and it is underestimated because replication attempts seldom occur and failures are rarely published.   

David Rimm's report of melanoma antibody staining that identified the need for specific chemotherapy could not be replicated in his own laboratory, a failure attributed to contaminated commercial antibodies and costing millions of dollars in wasted research funding. A 2008 report claimed that fewer than half of 6,000 routinely used antibodies recognized only their specific targets, and a 2015 report (with 110 co-signatories) on commercial antibodies called them "wildly variable."

Bayer Healthcare researchers reported in 2011 that fewer than one-fourth of 67 drug target validation studies could be replicated, and that more than half of attempted validations resulted in project terminations. A much-referenced report in 2012 from biotech giant Amgen found that its scientists could not replicate 47 of 53 "landmark" cancer papers from high impact factor journals. An unpublished 2015 survey by the American Society for Cell Biology found that more than two-thirds of responding researchers had experienced inability to replicate their research findings, and an accompanying comment was: "In many laboratories, the incentives to be first can be stronger than the incentives to be right."

When more than 270 researchers collaborated to replicate 150 reports from the most respected psychology journals, they rejected 50 as too difficult to replicate and found concordant outcomes in just 36 of the remaining 100 studies. The mean effect size of the replications was just half of that in the original studies, confirming much weaker results. The even worse replication rate for any but these top-rated reports can only be imagined.

Ninety percent of surveyed preclinical researchers reported that a research replication crisis exists. More than 70 percent of 1,576 surveyed researchers have tried and failed to replicate other scientists' experiment(s), and more than half of those surveyed reported failure(s) to reproduce their own experiments. Investigations of high-quality research results from leading academic institutions have shown a pathetic 11-25 percent replication rate for published preclinical research (here and here). The economic impact has been estimated at $28 billion annually for the more than half of preclinical research results that cannot be replicated.

John Ioannidis' claim that most science research results are false, controversial when it appeared in 2005, is now widely accepted. That the problem has nonetheless grown over the ensuing years is attributable to resistance from the research enterprise toward a reality that threatens their very existence. The status quo seems more important than the truth when careers and funding are at stake.

The many reasons for the failure to replicate most animal research results include a litany of laboratory procedural issues and the inherent variability of research involving nonhuman animals. In such an environment, the unethical practice of selective reporting of random successful outcomes is tolerated—often silently or overtly facilitated by institutional, career, and funding considerations that favor provocative claims over good science.

Critically however, underlying all this is the fundamental genetic and evolutionary reason for animal research irreproducibility and the failed translation of animal research findings to human medicine, which is immutable even if all other factors are eliminated—nonhumans cannot even predict or reproduce outcomes for their own species, never mind for humans.

Would better animal research replication mean better translation to human medicine? No, because no matter how reproducible or consistent animal research findings may be, those findings will not predict or explain human outcomes. Then why do we care about laboratory replication, other than presenting an honest picture to the public? We care because the fundamental inability even to replicate animal research outcomes contributes to the abundant evidence that such research in the interest of human medicine is scientifically irrelevant.

In other words, although we know that any animal testing results will have no reliable scientific link to human outcomes, the inability to conduct this bad science in a reproducible manner means that even the exercise itself has no scientific foundation. In even simpler words, the Emperor has no clothes. 

The replication crisis includes the animal research that often leads to FDA approval for clinical trials that—because of false preclinical results—cannot succeed. Thus, patients and volunteers are subjected to fruitless interventions and their consequences, years are lost to this doomed research, billions of dollars are wasted that may have been put to good scientific use, people die perhaps needlessly while bad research persists, and the once-hallowed scientific endeavor suffers a justified major credibility hit.

The NIH has recognized the replication crisis and suggested reasons and solutions. NIH director Francis Collins has stated: “Preclinical research, especially work that uses animal models, seems to be the area that is currently most susceptible to reproducibility issues.” So the crisis in animal research replication is no secret.

The proposed NIH approach is described:

"Clearly, reproducibility is not a problem that the NIH can tackle alone. Consequently, we are reaching out broadly to the research community, scientific publishers, universities, industry, professional organizations, patient-advocacy groups and other stakeholders to take the steps necessary to reset the self-corrective process of scientific inquiry."

Noble-sounding as it is, this is a formula for failure, as the elements solicited for resolution are precisely those who created and perpetuate the crisis. Self-interest mitigates against a cooperative effort, just as self-interest sustains an animal research enterprise that is an abject failure scientifically and ethically. The solution to the replication scandal does not appear to be at hand, but it can be eliminated by replacing animal research with human-relevant research methods.

A good single source review of the replication crisis in science research is here.