advertisement-vertical Download Proto magazine app
Social Icons
NOT AS EASY AS 1, 2, 3:
Groundbreaking research gets published in top journal // Replication study finds less promising result // Third study can’t reproduce results // If such studies are done at all

The Problem of Replication

By Linda Keslar // Photographs by Sam Kaplan
icon-pdfpdf icon-printprint
Framed, completed paint by numbers

Sam Kaplan

In 2012, when the journal Science published a study by a group of Cleveland researchers touting a cancer drug’s success against Alzheimer’s disease in mice, it looked like a potential breakthrough in a field long marked by setbacks and failures. As many as 5.4 million Americans have Alzheimer’s, and by one estimate, there’s a new case diagnosed every 69 seconds. While existing treatments can alleviate symptoms, there’s nothing to stop the progression of the ultimately fatal disease.

But in this study, researchers from Case Western Reserve University School of Medicine described how a drug called bexarotene appeared to clear the buildup of amyloid-beta plaques—protein deposits in the brain that are hallmarks of Alzheimer’s—in mice genetically engineered to have a condition similar to the human disease. The drug also seemed to improve the mice’s memory, cognitive abilities and social behavior. “The plaque reduction was an astounding finding,” says Sangram Sisodia, director of the Center for Molecular Neurobiology at the University of Chicago.

No other drug had attacked the plaques so rapidly. Bexarotene, marketed under the brand name Targretin, belongs to the retinoid class of drugs, related to vitamin A, that had been approved by the Food and Drug Administration in 1999 to treat a type of skin cancer called cutaneous T-cell lymphoma. According to the study in Science, bexarotene activated a gene that boosts production of apolipoprotein E, or ApoE, which can help break down amyloid-beta. Alzheimer’s patients can’t produce enough ApoE to prevent the protein from creating deposits. The drug also seemed to rouse a cellular mechanism that consumes the plaques.

Sisodia was one of many scientists who rushed to replicate the original study. “I think we all went back to our labs and tried to confirm these promising findings by repeating the initial experiments,” he says. Rudolph Tanzi, director of the genetics and aging research unit at Massachusetts General Hospital, also sought to repeat the research, spurred in part by physicians whose patients had heard about the apparent breakthrough and wanted to start taking bexarotene immediately. That was such a common response that in August 2012, The New England Journal of Medicine published an article warning physicians to wait for evidence from human clinical trials. “The results in mice certainly didn’t guarantee the drug would work in people,” says Tanzi. “And although it’s safe for cancer treatment, it can have severe side effects.”

But before testing whether the drug could help human Alzheimer’s patients, scientists needed to make sure it worked in mice—and while some labs were able to reproduce the drug’s effect on memory and cognition, the plaque reduction effect couldn’t be replicated. In four technical comments published in Science last May, several teams of independent researchers, including those co-authored by Sisodia and Tanzi, reported that their replication attempts showed no effect on plaque levels in lab mice treated with bexarotene. Months later, in August, a paper in the journal Molecular Neurodegeneration reported that researchers from Johns Hopkins had also failed to replicate either the plaque reduction or memory and cognition effects found in the Case Western research.

previous // next
icon-pdfpdf icon-printprint

1. “Why Most Published Research Findings Are False,” by John Ioannidis, PLOS Medicine, August 2005. In this seminal study on replication, Ioannidis, a Stanford University epidemiologist, uses statistical models and other key factors to demonstrate why the vast majority of published research findings are false.

2. “Believe It or Not: How Much Can We Rely on Published Data on Potential Drug Targets?” by F. Prinz, T. Schlange and K. Asadullah, Nature Reviews Drug Discovery, September 2011. In an industry analysis, three Bayer HealthCare scientists report that in-house experiments over four years failed to replicate two-thirds of 67 research studies in the fields of oncology, women’s health and cardiovascular diseases.

3. “Drug Development: Raise Standards for Preclinical Cancer Research,” by C.G. Begley and L. Ellis, Nature, March 29, 2012. A chronicle of scientists at American drug company Amgen who tried to replicate 53 studies they considered landmarks in the basic science of cancer—and were able to replicate only six.

Protomag on Facebook Protomag on Twitter