The rules of the game called psychological science

M. Bakker, A. van Dijk, J.M. Wicherts

Research output: Contribution to journalArticleScientificpeer-review

39 Downloads (Pure)

Abstract

If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies. Keywords: replication, sample size, power, publication bias, false positives
Original languageEnglish
Pages (from-to)543-554
JournalPerspectives on Psychological Science
Volume7
Issue number6
DOIs
Publication statusPublished - 2012

Fingerprint

Publication Bias

Cite this

@article{81dd3848e96a4dee9e3713d944e2c97c,
title = "The rules of the game called psychological science",
abstract = "If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96{\%} of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies. Keywords: replication, sample size, power, publication bias, false positives",
author = "M. Bakker and {van Dijk}, A. and J.M. Wicherts",
year = "2012",
doi = "10.1177/1745691612459060",
language = "English",
volume = "7",
pages = "543--554",
journal = "Perspectives on Psychological Science",
issn = "1745-6916",
publisher = "SAGE Publications Inc.",
number = "6",

}

The rules of the game called psychological science. / Bakker, M.; van Dijk, A.; Wicherts, J.M.

In: Perspectives on Psychological Science, Vol. 7, No. 6, 2012, p. 543-554.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - The rules of the game called psychological science

AU - Bakker, M.

AU - van Dijk, A.

AU - Wicherts, J.M.

PY - 2012

Y1 - 2012

N2 - If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies. Keywords: replication, sample size, power, publication bias, false positives

AB - If science were a game, a dominant rule would probably be to collect results that are statistically significant. Several reviews of the psychological literature have shown that around 96% of papers involving the use of null hypothesis significance testing report significant outcomes for their main results but that the typical studies are insufficiently powerful for such a track record. We explain this paradox by showing that the use of several small underpowered samples often represents a more efficient research strategy (in terms of finding p < .05) than does the use of one larger (more powerful) sample. Publication bias and the most efficient strategy lead to inflated effects and high rates of false positives, especially when researchers also resorted to questionable research practices, such as adding participants after intermediate testing. We provide simulations that highlight the severity of such biases in meta-analyses. We consider 13 meta-analyses covering 281 primary studies in various fields of psychology and find indications of biases and/or an excess of significant results in seven. These results highlight the need for sufficiently powerful replications and changes in journal policies. Keywords: replication, sample size, power, publication bias, false positives

U2 - 10.1177/1745691612459060

DO - 10.1177/1745691612459060

M3 - Article

VL - 7

SP - 543

EP - 554

JO - Perspectives on Psychological Science

JF - Perspectives on Psychological Science

SN - 1745-6916

IS - 6

ER -