Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the between-study variance in meta-analysis

Robbie C.m Van Aert*, Marcel A.l.m. Van Assen, Wolfgang Viechtbauer

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

7 Downloads (Pure)

Abstract

The effect sizes of studies included in a meta‐analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so‐called between‐study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between‐study variance facilitates interpretation of the meta‐analytic results. Two methods that are recommended to be used for creating such a confidence interval are the Q‐profile and generalized Q‐statistic method that both make use of the Q‐statistic. These methods are exact if the assumptions underlying the random‐effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte‐Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta‐analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random‐effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.
Original languageEnglish
Pages (from-to)225-239
JournalResearch Synthesis Methods
Volume10
Issue number2
DOIs
Publication statusPublished - 2019

Fingerprint

Effect Size
Statistical property
Statistic
Confidence interval
Metaanalysis
Coverage Probability
Exact Confidence Interval
Point Estimate
Sampling Distribution
Odds Ratio
Categorical or nominal
Gaussian distribution
Sample Size
Coverage
Monte Carlo Simulation
Simulation Study
Model
Estimate

Keywords

  • CLINICAL-TRIALS
  • FRAMEWORK
  • HETEROGENEITY
  • MOMENT-BASED ESTIMATORS
  • RANDOM-EFFECTS MODEL
  • confidence intervals
  • heterogeneity
  • meta-analysis
  • random-effects model

Cite this

@article{be607ef600c94623a312c373135412fb,
title = "Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the between-study variance in meta-analysis",
abstract = "The effect sizes of studies included in a meta‐analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so‐called between‐study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between‐study variance facilitates interpretation of the meta‐analytic results. Two methods that are recommended to be used for creating such a confidence interval are the Q‐profile and generalized Q‐statistic method that both make use of the Q‐statistic. These methods are exact if the assumptions underlying the random‐effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte‐Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta‐analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random‐effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.",
keywords = "CLINICAL-TRIALS, FRAMEWORK, HETEROGENEITY, MOMENT-BASED ESTIMATORS, RANDOM-EFFECTS MODEL, confidence intervals, heterogeneity, meta-analysis, random-effects model",
author = "{Van Aert}, {Robbie C.m} and {Van Assen}, {Marcel A.l.m.} and Wolfgang Viechtbauer",
year = "2019",
doi = "10.1002/jrsm.1336",
language = "English",
volume = "10",
pages = "225--239",
journal = "Research Synthesis Methods",
issn = "1759-2879",
number = "2",

}

Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the between-study variance in meta-analysis. / Van Aert, Robbie C.m; Van Assen, Marcel A.l.m.; Viechtbauer, Wolfgang.

In: Research Synthesis Methods, Vol. 10, No. 2, 2019, p. 225-239.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Statistical properties of methods based on the Q-statistic for constructing a confidence interval for the between-study variance in meta-analysis

AU - Van Aert, Robbie C.m

AU - Van Assen, Marcel A.l.m.

AU - Viechtbauer, Wolfgang

PY - 2019

Y1 - 2019

N2 - The effect sizes of studies included in a meta‐analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so‐called between‐study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between‐study variance facilitates interpretation of the meta‐analytic results. Two methods that are recommended to be used for creating such a confidence interval are the Q‐profile and generalized Q‐statistic method that both make use of the Q‐statistic. These methods are exact if the assumptions underlying the random‐effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte‐Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta‐analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random‐effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.

AB - The effect sizes of studies included in a meta‐analysis do often not share a common true effect size due to differences in for instance the design of the studies. Estimates of this so‐called between‐study variance are usually imprecise. Hence, reporting a confidence interval together with a point estimate of the amount of between‐study variance facilitates interpretation of the meta‐analytic results. Two methods that are recommended to be used for creating such a confidence interval are the Q‐profile and generalized Q‐statistic method that both make use of the Q‐statistic. These methods are exact if the assumptions underlying the random‐effects model hold, but these assumptions are usually violated in practice such that confidence intervals of the methods are approximate rather than exact confidence intervals. We illustrate by means of two Monte‐Carlo simulation studies with odds ratio as effect size measure that coverage probabilities of both methods can be substantially below the nominal coverage rate in situations that are representative for meta‐analyses in practice. We also show that these too low coverage probabilities are caused by violations of the assumptions of the random‐effects model (ie, normal sampling distributions of the effect size measure and known sampling variances) and are especially prevalent if the sample sizes in the primary studies are small.

KW - CLINICAL-TRIALS

KW - FRAMEWORK

KW - HETEROGENEITY

KW - MOMENT-BASED ESTIMATORS

KW - RANDOM-EFFECTS MODEL

KW - confidence intervals

KW - heterogeneity

KW - meta-analysis

KW - random-effects model

U2 - 10.1002/jrsm.1336

DO - 10.1002/jrsm.1336

M3 - Article

VL - 10

SP - 225

EP - 239

JO - Research Synthesis Methods

JF - Research Synthesis Methods

SN - 1759-2879

IS - 2

ER -