Enhancing Rigor and Reproducibility

In recent years, there has been a growing awareness of the need for rigorously designed published preclinical studies, to ensure that research findings can be faithfully replicated. In response to this need, the National Institutes of Health (NIH) has established a formal initiative aimed at improving research reproducibility and are implementing significant changes to their policies, grant applications and peer-review process, which took effect Jan. 25, 2016.

UC Irvine is committed to upholding the highest standards of conducting biomedical research and Dr. Oswald Steward, senior associate dean for research in the School of Medicine, is leading a campus-wide effort to ensure that the UC Irvine research community is informed about these important changes and have resources to address the "reproducibly problem" proactively.

This webpage serves as a means to communicate important updates from NIH and post resources to help UC Irvine researchers advance their own research with rigor and transparency.

Resources for enhancing reproducibility through rigor and transparency

May 18, 2017: Workshop led by Dr. Oswald Steward

• View workshop video recording ›

This workshop is an update to the January 2017 workshop and provides information on how study sections are evaluating the 4 new required sections for scientific rigor and reproducibility.

Jan. 27, 2017: Workshop led by Drs. Oswald Steward, Christine Gall, and Alan Goldin

This workshop is an update to the December 2015 workshop and includes a panel of current study section members who provide information on how study sections are actually evaluating the 4 new required sections for scientific rigor and reproducibility.

• View workshop video recording ›
• View presentation slide deck ›

Dec. 11, 2015: Workshop led by Dr. Oswald Steward 

In this workshop, Dr. Steward presents a comprehensive overview of the "reproducibility problem," new requirements for NIH grant applications, and suggested best practices to meet the new standards for scientific rigor.

• View workshop video recording ›
• View presentation slide deck ›

NIH resources for enhancing reproducibility through rigor and transparency

• Peer Reviewer Criteria for Rigor and Reproducibility — This NIH presentation clarifies the role of peer reviewers when assessing scientific rigor in grant applications. NIH has also issued a decision tree for reviewers when evaluating sex as a biological variable.
  
  
• NIH Guidance and Examples of Reproducibility and Rigor in Grant Applications — NIH has published guidance to assist the extramural community in addressing rigor and reproducibility in grant applications as well as examples of rigorous experimental design from funded applications.
  
  
• NIH Principles & Guidelines — These principles and guidelines are the result of a 2014 workshop held by the NIH, Nature Publishing Group and Science, which focused on identifying common opportunities in the scientific publishing arena to enhance rigor and further support research that is reproducible, robust and transparent.
  
  
• NIH Publications — These are publications from NIH authors on the issue of reproducibility and NIH’s actions to enhance reproducibility. Topics include fixing cell lines and how to balance sex in cell and animal studies.
  
  
• NIH Training Modules — NIH developed four (4) video modules with accompanying discussion materials that focus on integral components of reproducibility and rigor in the research endeavor, such as bias, blinding and exclusion criteria. NIH also released a training module developed for NIH staff that provides a comprehensive overview of NIH revised policy, grant application instructions, and review criteria for rigor and reproducibility.
  
  
• NIH Meetings & Workshops — NIH has hosted several meetings and workshops focused on rigor, reproducibility, and transparency in scientific research and has posted materials from these events.
  
  
• Expanded Guidelines for Reporting Preclinical Research — These guidelines are from external organizations that use the NIH principles and guidelines as a model, but have been expanded to fit the unique needs and challenges of specific fields.
  
  
• NIH Revised Grant Application Instructions — This is an overview of the new instructions and revised review criteria, which will focus on four areas deemed important for enhancing rigor and transparency: (1) scientific premise of proposed research; (2) rigorous experimental design; (3) consideration of sex and other relevant biological variables; (4) authentication of key biological and/or chemical resources. There is also a revised version of the SF424 Application Instructions for applications due on or after Jan. 25, 2016.
  
  
• NIH FAQS about Rigor and Reproducibility — The NIH answers frequently asked questions about changes to its policy and grant applications.

Additional resources

Authentication of key biological and/or chemical resources

• Colleagues at UC San Diego have published examples of how to authenticate key resources such as cell lines and antibodies to include in NIH grant applications.

Articles about failure to replicate

• Prinz F, Schlange T, Asadullah K. Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery. 2011; 10(9):712. PMID 21892149.
  
  
• Begley CG, Ellis LM. Raise standards for preclinical cancer research. Nature. 2012; 483(7391):531-3. PMID 22460880.
  
  
• Steward O, Popovich PG, Dietrich WD, Kleitman N. Replication and reproducibility in spinal cord research. Exp. Neurol. 2012; 233(2):597-605. PMID 22078756.
  
  
• Begley, CG. Six red flags for suspect work. Nature. 2013; 497(745):433-4. PMID 23698428

Articles about sex as a biological variable

• Becker JB, Arnold AP, Berkley KJ, Blaustein JD, Eckel LA, Hampson E, Herman JP, Marts S, Sadee W, Steiner M, Taylor J, Young E. Strategies and Methods for Research on Sex Differences in Brain and Behavior. Endocrinology. 2005; 146(4):1650-73. PMID 15618360

• Ritz SA, Antle DM, Côté J, Deroy K, Fraleigh N, Messing K, Parent L, St-Pierre J, Vaillancourt C, Mergler D. First steps for integrating sex and gender considerations into basic experimental biomedical research. FASEB J. 2014; 28(1):4-13. PMID: 24056086

• McCullough LD, de Vries JG, Miller VM, Becker JB, Sandberg K, McCarthy MM. NIH initiative to balance sex of animals in preclinical studies: generative questions to guide policy, implementation, and metrics.  Biol Sex Differ. 2014; 5: 15. PMCID: PMC4360141

• Clayton JA. Studying both sexes: a guiding principle for biomedicine. FASEB J. 2015. pii: fj.15-279554. PMID 26514164

• Jill B. Becker JB, Prendergast BJ, Liang JW. Female rats are not more variable than male rats: a meta-analysis of neuroscience studies. Biol Sex Differ. 2016; 7:34. PMCID: PMC4962440

• Prendergast BJ, Onishi KG, Zucker I. Female mice liberated for inclusion in neuroscience and biomedical research. Neurosci Biobehav Rev. 2014; 40:1-5. PMID 24456941

Articles about best practices for rigor and reproducibility

• Insel, T. (2014) NIH Director’s Blog: P-Hacking 
  

• Landis SC, Amara SG, Asadullak K, Austin CP, Blumenstein R, Bradley EW, Crystal RG, Darnell RB, Ferrante RJ, Fillit H, Finkelstein R, Fisher M, Gendelman HE, Golub RM, Goudreau JL, Gross RA, Gubitz AK, Hesterlee SE, Howells DW, Huguenard J, Kelner K, Koroshetz W, Krainc D, Lazic SE, Levine MS, Macleod MR, McCall JM, Moxley RT, Narasimhan K, Noble LJ, Perrin S, Porter JD, Steward O, Unger E, Ultz U, Silberberg SD.  A call for transparent reporting to optimize the predictive value of preclinical research. Nature. 2012; 490(7419):187-91. PMID 23060188.PMC3511845
  
  
• Steward O, Balice-Gordon R. Rigor or mortis: best practices for preclinical research in neuroscience. Neuron. 2014; 84(3): 572-81. PMID 25442936