Ensuring the quality and specificity of preregistrations

Marjan Bakker*, Coosje L. S. Veldkamp, Marcel A. L. M. van Assen, Elise A. V. Crompvoets, How Hwee Ong, Brian A. Nosek, Courtney K. Soderberg, David Mellor, Jelte M. Wicherts, Lisa Bero (Editor)

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

Researchers face many, often seemingly arbitrary, choices in formulating hypotheses, designing protocols, collecting data, analyzing data, and reporting results. Opportunistic use of “researcher degrees of freedom” aimed at obtaining statistical significance increases the likelihood of obtaining and publishing false-positive results and overestimated effect sizes. Preregistration is a mechanism for reducing such degrees of freedom by specifying designs and analysis plans before observing the research outcomes. The effectiveness of preregistration may depend, in part, on whether the process facilitates sufficiently specific articulation of such plans. In this preregistered study, we compared 2 formats of preregistration available on the OSF: Standard Pre-Data Collection Registration and Prereg Challenge Registration (now called “OSF Preregistration,” http://osf.io/prereg/). The Prereg Challenge format was a “structured” workflow with detailed instructions and an independent review to confirm completeness; the “Standard” format was “unstructured” with minimal direct guidance to give researchers flexibility for what to prespecify. Results of comparing random samples of 53 preregistrations from each format indicate that the “structured” format restricted the opportunistic use of researcher degrees of freedom better (Cliff’s Delta = 0.49) than the “unstructured” format, but neither eliminated all researcher degrees of freedom. We also observed very low concordance among coders about the number of hypotheses (14%), indicating that they are often not clearly stated. We conclude that effective preregistration is challenging, and registration formats that provide effective guidance may improve the quality of research.
Original languageEnglish
Article numbere3000937
JournalPLOS Biology
Volume18
Issue number12
DOIs
Publication statusPublished - 2020

Keywords

  • CLINICAL-TRIALS
  • JOURNALS
  • OUTCOME REPORTING BIAS
  • POWER
  • QUESTIONABLE RESEARCH PRACTICES
  • RANDOMIZED-TRIALS
  • REGISTERED-REPORTS
  • REGISTRATION

Fingerprint Dive into the research topics of 'Ensuring the quality and specificity of preregistrations'. Together they form a unique fingerprint.

Cite this