TY - JOUR
T1 - Times are changing, bias isn't
T2 - A meta-meta-analysis on publication bias detection practices, prevalence rates, and predictors in industrial/organizational psychology
AU - Siegel, Magdalena
AU - Eder, Junia Sophia Nur
AU - Wicherts, Jelte M.
AU - Pietschnig, Jakob
N1 - The research by MS, JSNE, and JP was funded by the Austrian Science Fund (FWF) project number P31002. The research by JMW was supported by a Consolidator Grant 726361 (IMPROVE) from the European Research Council (ERC)
PY - 2022
Y1 - 2022
N2 - Effect misestimations plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (a) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (b) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (c) the prevalence of potential publication bias is concerning but mostly remains undetected. Although our results indicate somewhat of a trend toward higher bias awareness, they substantiate concerns about potential publication bias in I/O Psychology, warranting increased researcher awareness about appropriate and state-of-the-art bias detection and triangulation. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.
AB - Effect misestimations plague Psychological Science, but advances in the identification of dissemination biases in general and publication bias in particular have helped in dealing with biased effects in the literature. However, the application of publication bias detection methods appears to be not equally prevalent across subdisciplines. It has been suggested that particularly in I/O Psychology, appropriate publication bias detection methods are underused. In this meta-meta-analysis, we present prevalence estimates, predictors, and time trends of publication bias in 128 meta-analyses that were published in the Journal of Applied Psychology (7,263 effect sizes, 3,000,000+ participants). Moreover, we reanalyzed data of 87 meta-analyses and applied nine standard and more modern publication bias detection methods. We show that (a) the bias detection method applications are underused (only 41% of meta-analyses use at least one method) but have increased in recent years, (b) those meta-analyses that apply such methods now use more, but mostly inappropriate methods, and (c) the prevalence of potential publication bias is concerning but mostly remains undetected. Although our results indicate somewhat of a trend toward higher bias awareness, they substantiate concerns about potential publication bias in I/O Psychology, warranting increased researcher awareness about appropriate and state-of-the-art bias detection and triangulation. Embracing open science practices such as data sharing or study preregistration is needed to raise reproducibility and ultimately strengthen Psychological Science in general and I/O Psychology in particular.
KW - dissemination bias
KW - publication bias
KW - meta-meta-analysis
KW - industrial and organizational psychology
KW - decline effect
KW - WORK-FAMILY CONFLICT
KW - COGNITIVE-ABILITY TESTS
KW - MEMBER EXCHANGE LMX
KW - FILE DRAWER PROBLEM
KW - JOB-PERFORMANCE
KW - METAANALYTIC EXAMINATION
KW - 5-FACTOR MODEL
KW - COMPREHENSIVE METAANALYSIS
KW - ORGANIZATIONAL COMMITMENT
KW - CONDUCTING METAANALYSES
UR - http://www.scopus.com/inward/record.url?scp=85123214390&partnerID=8YFLogxK
U2 - 10.1037/apl0000991
DO - 10.1037/apl0000991
M3 - Article
SN - 0021-9010
VL - 107
SP - 2013
EP - 2039
JO - Journal of Applied Psychology
JF - Journal of Applied Psychology
IS - 11
ER -