Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies

A.O.J. Cramer, Don van Ravenzwaaij, Dora Matzke, Helen Steingroever, Ruud Wetzels, Raoul P P P Grasman, Lourens J Waldorp, Eric-Jan Wagenmakers

Research output: Contribution to journalArticleScientificpeer-review

5 Downloads (Pure)

Abstract

Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

Original languageEnglish
Pages (from-to)640-647
JournalPsychonomic Bulletin & Review
Volume23
Issue number2
DOIs
Publication statusPublished - 2016
Externally publishedYes

Fingerprint

Multiplicity
Analysis of Variance
Null Hypothesis
Harbors
Psychologists
Interaction

Keywords

  • Analysis of Variance
  • Biomedical Research
  • Data Interpretation, Statistical
  • Humans
  • Psychology
  • Journal Article

Cite this

Cramer, A. O. J., van Ravenzwaaij, D., Matzke, D., Steingroever, H., Wetzels, R., Grasman, R. P. P. P., ... Wagenmakers, E-J. (2016). Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies. Psychonomic Bulletin & Review, 23(2), 640-647. https://doi.org/10.3758/s13423-015-0913-5
Cramer, A.O.J. ; van Ravenzwaaij, Don ; Matzke, Dora ; Steingroever, Helen ; Wetzels, Ruud ; Grasman, Raoul P P P ; Waldorp, Lourens J ; Wagenmakers, Eric-Jan. / Hidden multiplicity in exploratory multiway ANOVA : Prevalence and remedies. In: Psychonomic Bulletin & Review. 2016 ; Vol. 23, No. 2. pp. 640-647.
@article{aac42f2468ac487c892dc54f0d4958c1,
title = "Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies",
abstract = "Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 {\%} rather than 5 {\%}, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.",
keywords = "Analysis of Variance, Biomedical Research, Data Interpretation, Statistical, Humans, Psychology, Journal Article",
author = "A.O.J. Cramer and {van Ravenzwaaij}, Don and Dora Matzke and Helen Steingroever and Ruud Wetzels and Grasman, {Raoul P P P} and Waldorp, {Lourens J} and Eric-Jan Wagenmakers",
year = "2016",
doi = "10.3758/s13423-015-0913-5",
language = "English",
volume = "23",
pages = "640--647",
journal = "Psychonomic Bulletin & Review",
issn = "1069-9384",
publisher = "Springer",
number = "2",

}

Cramer, AOJ, van Ravenzwaaij, D, Matzke, D, Steingroever, H, Wetzels, R, Grasman, RPPP, Waldorp, LJ & Wagenmakers, E-J 2016, 'Hidden multiplicity in exploratory multiway ANOVA: Prevalence and remedies', Psychonomic Bulletin & Review, vol. 23, no. 2, pp. 640-647. https://doi.org/10.3758/s13423-015-0913-5

Hidden multiplicity in exploratory multiway ANOVA : Prevalence and remedies. / Cramer, A.O.J.; van Ravenzwaaij, Don; Matzke, Dora; Steingroever, Helen; Wetzels, Ruud; Grasman, Raoul P P P; Waldorp, Lourens J; Wagenmakers, Eric-Jan.

In: Psychonomic Bulletin & Review, Vol. 23, No. 2, 2016, p. 640-647.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Hidden multiplicity in exploratory multiway ANOVA

T2 - Prevalence and remedies

AU - Cramer, A.O.J.

AU - van Ravenzwaaij, Don

AU - Matzke, Dora

AU - Steingroever, Helen

AU - Wetzels, Ruud

AU - Grasman, Raoul P P P

AU - Waldorp, Lourens J

AU - Wagenmakers, Eric-Jan

PY - 2016

Y1 - 2016

N2 - Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

AB - Many psychologists do not realize that exploratory use of the popular multiway analysis of variance harbors a multiple-comparison problem. In the case of two factors, three separate null hypotheses are subject to test (i.e., two main effects and one interaction). Consequently, the probability of at least one Type I error (if all null hypotheses are true) is 14 % rather than 5 %, if the three tests are independent. We explain the multiple-comparison problem and demonstrate that researchers almost never correct for it. To mitigate the problem, we describe four remedies: the omnibus F test, control of the familywise error rate, control of the false discovery rate, and preregistration of the hypotheses.

KW - Analysis of Variance

KW - Biomedical Research

KW - Data Interpretation, Statistical

KW - Humans

KW - Psychology

KW - Journal Article

U2 - 10.3758/s13423-015-0913-5

DO - 10.3758/s13423-015-0913-5

M3 - Article

C2 - 26374437

VL - 23

SP - 640

EP - 647

JO - Psychonomic Bulletin & Review

JF - Psychonomic Bulletin & Review

SN - 1069-9384

IS - 2

ER -