Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

Research output: Contribution to journalArticleScientificpeer-review

1 Downloads (Pure)

Abstract

Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

Original languageEnglish
Article number0215052
Pages (from-to)e0215052
Number of pages32
JournalPLoS ONE
Volume14
Issue number4
DOIs
Publication statusPublished - 2019

Fingerprint

Publication Bias
Medicine
Literature
Consensus
Monte Carlo simulation
Databases

Keywords

  • BETWEEN-STUDY HETEROGENEITY
  • CONDUCTING METAANALYSES
  • ESTIMATING EFFECT SIZE
  • FILL METHOD
  • FUNNEL-PLOT
  • JUDGMENT CALLS
  • PREDICTIVE-DISTRIBUTIONS
  • PREVALENCE
  • SCIENCE
  • STATISTICAL TESTS

Cite this

@article{47c004f5a5d6485ca2a2d1fd233159ad,
title = "Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis",
abstract = "Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8{\%} of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5{\%} statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9{\%} and 18.9{\%} for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.",
keywords = "BETWEEN-STUDY HETEROGENEITY, CONDUCTING METAANALYSES, ESTIMATING EFFECT SIZE, FILL METHOD, FUNNEL-PLOT, JUDGMENT CALLS, PREDICTIVE-DISTRIBUTIONS, PREVALENCE, SCIENCE, STATISTICAL TESTS",
author = "{Van Aert}, {Robbie C. M.} and Wicherts, {Jelte M.} and {Van Assen}, {Marcel A. L. M.}",
year = "2019",
doi = "10.1371/journal.pone.0215052",
language = "English",
volume = "14",
pages = "e0215052",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "PUBLIC LIBRARY SCIENCE",
number = "4",

}

Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis. / Van Aert, Robbie C. M.; Wicherts, Jelte M.; Van Assen, Marcel A. L. M.

In: PLoS ONE, Vol. 14, No. 4, 0215052, 2019, p. e0215052.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Publication bias examined in meta-analyses from psychology and medicine: A meta-meta-analysis

AU - Van Aert, Robbie C. M.

AU - Wicherts, Jelte M.

AU - Van Assen, Marcel A. L. M.

PY - 2019

Y1 - 2019

N2 - Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

AB - Publication bias is a substantial problem for the credibility of research in general and of meta-analyses in particular, as it yields overestimated effects and may suggest the existence of non-existing effects. Although there is consensus that publication bias exists, how strongly it affects different scientific literatures is currently less well-known. We examined evidence of publication bias in a large-scale data set of primary studies that were included in 83 meta-analyses published in Psychological Bulletin (representing meta-analyses from psychology) and 499 systematic reviews from the Cochrane Database of Systematic Reviews (CDSR; representing meta-analyses from medicine). Publication bias was assessed on all homogeneous subsets (3.8% of all subsets of meta-analyses published in Psychological Bulletin) of primary studies included in meta-analyses, because publication bias methods do not have good statistical properties if the true effect size is heterogeneous. Publication bias tests did not reveal evidence for bias in the homogeneous subsets. Overestimation was minimal but statistically significant, providing evidence of publication bias that appeared to be similar in both fields. However, a Monte-Carlo simulation study revealed that the creation of homogeneous subsets resulted in challenging conditions for publication bias methods since the number of effect sizes in a subset was rather small (median number of effect sizes equaled 6). Our findings are in line with, in its most extreme case, publication bias ranging from no bias until only 5% statistically nonsignificant effect sizes being published. These and other findings, in combination with the small percentages of statistically significant primary effect sizes (28.9% and 18.9% for subsets published in Psychological Bulletin and CDSR), led to the conclusion that evidence for publication bias in the studied homogeneous subsets is weak, but suggestive of mild publication bias in both psychology and medicine.

KW - BETWEEN-STUDY HETEROGENEITY

KW - CONDUCTING METAANALYSES

KW - ESTIMATING EFFECT SIZE

KW - FILL METHOD

KW - FUNNEL-PLOT

KW - JUDGMENT CALLS

KW - PREDICTIVE-DISTRIBUTIONS

KW - PREVALENCE

KW - SCIENCE

KW - STATISTICAL TESTS

U2 - 10.1371/journal.pone.0215052

DO - 10.1371/journal.pone.0215052

M3 - Article

VL - 14

SP - e0215052

JO - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 4

M1 - 0215052

ER -