The effect of publication bias on the Q test and assessment of heterogeneity

Hilde E M Augusteijn*, Robbie C M van Aert, M.A.L.M. van Assen

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

Abstract

One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Original languageEnglish
Pages (from-to)116-134
JournalPsychological Methods
Volume24
Issue number1
DOIs
Publication statusPublished - 2019

Fingerprint

Publication Bias

Keywords

  • DECISIONS
  • DISTRIBUTIONS
  • EFFECT SIZE
  • METAANALYSIS
  • P VALUES
  • Q test
  • RECOMMENDATIONS
  • VARIANCE ESTIMATORS
  • extreme homogeneity
  • heterogeneity
  • meta-analysis
  • publication bias

Cite this

@article{3d1ea79b94a84c09bcefc91496148268,
title = "The effect of publication bias on the Q test and assessment of heterogeneity",
abstract = "One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).",
keywords = "DECISIONS, DISTRIBUTIONS, EFFECT SIZE, METAANALYSIS, P VALUES, Q test, RECOMMENDATIONS, VARIANCE ESTIMATORS, extreme homogeneity, heterogeneity, meta-analysis, publication bias",
author = "Augusteijn, {Hilde E M} and {van Aert}, {Robbie C M} and {van Assen}, M.A.L.M.",
year = "2019",
doi = "10.1037/met0000197",
language = "English",
volume = "24",
pages = "116--134",
journal = "Psychological Methods",
issn = "1082-989X",
publisher = "AMER PSYCHOLOGICAL ASSOC",
number = "1",

}

The effect of publication bias on the Q test and assessment of heterogeneity. / Augusteijn, Hilde E M; van Aert, Robbie C M; van Assen, M.A.L.M.

In: Psychological Methods, Vol. 24, No. 1, 2019, p. 116-134.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - The effect of publication bias on the Q test and assessment of heterogeneity

AU - Augusteijn, Hilde E M

AU - van Aert, Robbie C M

AU - van Assen, M.A.L.M.

PY - 2019

Y1 - 2019

N2 - One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

AB - One of the main goals of meta-analysis is to test for and estimate the heterogeneity of effect sizes. We examined the effect of publication bias on the Q test and assessments of heterogeneity as a function of true heterogeneity, publication bias, true effect size, number of studies, and variation of sample sizes. The present study has two main contributions and is relevant to all researchers conducting meta-analysis. First, we show when and how publication bias affects the assessment of heterogeneity. The expected values of heterogeneity measures H² and I² were analytically derived, and the power and Type I error rate of the Q test were examined in a Monte Carlo simulation study. Our results show that the effect of publication bias on the Q test and assessment of heterogeneity is large, complex, and nonlinear. Publication bias can both dramatically decrease and increase heterogeneity in true effect size, particularly if the number of studies is large and population effect size is small. We therefore conclude that the Q test of homogeneity and heterogeneity measures H² and I² are generally not valid when publication bias is present. Our second contribution is that we introduce a web application, Q-sense, which can be used to determine the impact of publication bias on the assessment of heterogeneity within a certain meta-analysis and to assess the robustness of the meta-analytic estimate to publication bias. Furthermore, we apply Q-sense to 2 published meta-analyses, showing how publication bias can result in invalid estimates of effect size and heterogeneity. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

KW - DECISIONS

KW - DISTRIBUTIONS

KW - EFFECT SIZE

KW - METAANALYSIS

KW - P VALUES

KW - Q test

KW - RECOMMENDATIONS

KW - VARIANCE ESTIMATORS

KW - extreme homogeneity

KW - heterogeneity

KW - meta-analysis

KW - publication bias

U2 - 10.1037/met0000197

DO - 10.1037/met0000197

M3 - Article

VL - 24

SP - 116

EP - 134

JO - Psychological Methods

JF - Psychological Methods

SN - 1082-989X

IS - 1

ER -