Researchers' intuitions about power in psychological research

M. Bakker, C. H. J. Hartgerink, J. M. Wicherts, H. L. J. Van Der Maas

Research output: Contribution to journalArticleScientificpeer-review

38 Downloads (Pure)

Abstract

Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.
Keywords: power, survey, methodology, sample size, effect size, open data, open materials
Original languageEnglish
Pages (from-to)1069-1077
JournalPsychological Science
Volume27
Issue number8
DOIs
Publication statusPublished - 2016

Fingerprint

Intuition
Cell Size
Surveys and Questionnaires

Cite this

Bakker, M. ; Hartgerink, C. H. J. ; Wicherts, J. M. ; Van Der Maas, H. L. J. / Researchers' intuitions about power in psychological research. In: Psychological Science. 2016 ; Vol. 27, No. 8. pp. 1069-1077.
@article{3efd6f7418b947ca9d4848cc4af3aed4,
title = "Researchers' intuitions about power in psychological research",
abstract = "Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89{\%} of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95{\%} underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.Keywords: power, survey, methodology, sample size, effect size, open data, open materials",
author = "M. Bakker and Hartgerink, {C. H. J.} and Wicherts, {J. M.} and {Van Der Maas}, {H. L. J.}",
year = "2016",
doi = "10.1177/0956797616647519",
language = "English",
volume = "27",
pages = "1069--1077",
journal = "Psychological Science",
issn = "0956-7976",
publisher = "Sage Publications, Inc.",
number = "8",

}

Researchers' intuitions about power in psychological research. / Bakker, M.; Hartgerink, C. H. J.; Wicherts, J. M.; Van Der Maas, H. L. J.

In: Psychological Science, Vol. 27, No. 8, 2016, p. 1069-1077.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Researchers' intuitions about power in psychological research

AU - Bakker, M.

AU - Hartgerink, C. H. J.

AU - Wicherts, J. M.

AU - Van Der Maas, H. L. J.

PY - 2016

Y1 - 2016

N2 - Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.Keywords: power, survey, methodology, sample size, effect size, open data, open materials

AB - Many psychology studies are statistically underpowered. In part, this may be because many researchers rely on intuition, rules of thumb, and prior practice (along with practical considerations) to determine the number of subjects to test. In Study 1, we surveyed 291 published research psychologists and found large discrepancies between their reports of their preferred amount of power and the actual power of their studies (calculated from their reported typical cell size, typical effect size, and acceptable alpha). Furthermore, in Study 2, 89% of the 214 respondents overestimated the power of specific research designs with a small expected effect size, and 95% underestimated the sample size needed to obtain .80 power for detecting a small effect. Neither researchers’ experience nor their knowledge predicted the bias in their self-reported power intuitions. Because many respondents reported that they based their sample sizes on rules of thumb or common practice in the field, we recommend that researchers conduct and report formal power analyses for their studies.Keywords: power, survey, methodology, sample size, effect size, open data, open materials

U2 - 10.1177/0956797616647519

DO - 10.1177/0956797616647519

M3 - Article

VL - 27

SP - 1069

EP - 1077

JO - Psychological Science

JF - Psychological Science

SN - 0956-7976

IS - 8

ER -