Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size

Marjan Bakker*, Coosje L.S. Veldkamp, Olmo R. van den Akker, Marcel A.L.M. van Assen, Elise Crompvoets, How Hwee Ong, Jelte M. Wicherts

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

250 Downloads (Pure)

Abstract

In this preregistered study, we investigated whether the statistical power of a study is higher when researchers are asked to make a formal power analysis before collecting data. We compared the sample size descriptions from two sources: (i) a sample of pre-registrations created according to the guidelines for the Center for Open Science Preregistration Challenge (PCRs) and a sample of institutional review board (IRB) proposals from Tilburg School of Behavior and Social Sciences, which both include a recommendation to do a formal power analysis, and (ii) a sample of pre-registrations created according to the guidelines for Open Science Framework Standard Pre-Data Collection Registrations (SPRs) in which no guidance on sample size planning is given. We found that PCRs and IRBs (72%) more often included sample size decisions based on power analyses than the SPRs (45%). However, this did not result in larger planned sample sizes. The determined sample size of the PCRs and IRB proposals (Md = 90.50) was not higher than the determined sample size of the SPRs (Md = 126.00; W = 3389.5, p = 0.936). Typically, power analyses in the registrations were conducted with G*power, assuming a medium effect size, α = .05 and a power of .80. Only 20% of the power analyses contained enough information to fully reproduce the results and only 62% of these power analyses pertained to the main hypothesis test in the pre-registration. Therefore, we see ample room for improvements in the quality of the registrations and we offer several recommendations to do so.

Original languageEnglish
Article numbere0236079
JournalPLoS ONE
Volume15
Issue number7
DOIs
Publication statusPublished - 2020

Keywords

  • Ethics Committees, Research
  • Sample Size
  • Statistics as Topic/methods

Fingerprint Dive into the research topics of 'Recommendations in pre-registrations and internal review board proposals promote formal power analyses but do not increase sample size'. Together they form a unique fingerprint.

  • Cite this