Distributions of p-values smaller than .05 in psychology

What is going on?

Research output: Contribution to journalArticleScientificpeer-review

84 Downloads (Pure)

Abstract

Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP "incorrect rounding of p-value" (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.
Original languageEnglish
Article numbere1935
JournalPEERJ
Volume4
DOIs
Publication statusPublished - 2016

Keywords

  • p-values
  • NHST
  • QRP
  • Caliper test
  • Data peeking

Cite this

@article{b258553a4db44e2a903969ccc610b836,
title = "Distributions of p-values smaller than .05 in psychology: What is going on?",
abstract = "Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP {"}incorrect rounding of p-value{"} (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.",
keywords = "p-values, NHST, QRP, Caliper test, Data peeking",
author = "C.H.J. Hartgerink and {van Aert}, R.C.M. and M.B. Nuijten and J.M. Wicherts and {van Assen}, M.A.L.M.",
year = "2016",
doi = "10.7717/peerj.1935",
language = "English",
volume = "4",
journal = "PEERJ",
issn = "2167-8359",
publisher = "PeerJ",

}

Distributions of p-values smaller than .05 in psychology : What is going on? / Hartgerink, C.H.J.; van Aert, R.C.M.; Nuijten, M.B.; Wicherts, J.M.; van Assen, M.A.L.M.

In: PEERJ, Vol. 4, e1935, 2016.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Distributions of p-values smaller than .05 in psychology

T2 - What is going on?

AU - Hartgerink, C.H.J.

AU - van Aert, R.C.M.

AU - Nuijten, M.B.

AU - Wicherts, J.M.

AU - van Assen, M.A.L.M.

PY - 2016

Y1 - 2016

N2 - Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP "incorrect rounding of p-value" (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.

AB - Previous studies provided mixed findings on pecularities in p-value distributions in psychology. This paper examined 258,050 test results across 30,710 articles from eight high impact journals to investigate the existence of a peculiar prevalence of p-values just below .05 (i.e., a bump) in the psychological literature, and a potential increase thereof over time. We indeed found evidence for a bump just below .05 in the distribution of exactly reported p-values in the journals Developmental Psychology, Journal of Applied Psychology, and Journal of Personality and Social Psychology, but the bump did not increase over the years and disappeared when using recalculated p-values. We found clear and direct evidence for the QRP "incorrect rounding of p-value" (John, Loewenstein & Prelec, 2012) in all psychology journals. Finally, we also investigated monotonic excess of p-values, an effect of certain QRPs that has been neglected in previous research, and developed two measures to detect this by modeling the distributions of statistically significant p-values. Using simulations and applying the two measures to the retrieved test results, we argue that, although one of the measures suggests the use of QRPs in psychology, it is difficult to draw general conclusions concerning QRPs based on modeling of p-value distributions.

KW - p-values

KW - NHST

KW - QRP

KW - Caliper test

KW - Data peeking

U2 - 10.7717/peerj.1935

DO - 10.7717/peerj.1935

M3 - Article

VL - 4

JO - PEERJ

JF - PEERJ

SN - 2167-8359

M1 - e1935

ER -