A power fallacy

Eric-Jan Wagenmakers, Josine Verhagen, Alexander Ly, M. Bakker, Michael D Lee, Dora Matzke, Jeffrey N Rouder, Richard D Morey

Research output: Contribution to journalArticleScientificpeer-review

Abstract

The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a particular outcome from a high-power experiment can be completely uninformative, whereas a particular outcome from a low-power experiment can be highly informative. Although power is useful in planning an experiment, it is less useful-and sometimes even misleading-for making inferences from observed data. To make inferences from data, we recommend the use of likelihood ratios or Bayes factors, which are the extension of likelihood ratios beyond point hypotheses. These methods of inference do not average over hypothetical replications of an experiment, but instead condition on the data that have actually been observed. In this way, likelihood ratios and Bayes factors rationally quantify the evidence that a particular data set provides for or against the null or any other hypothesis.

Original languageEnglish
Pages (from-to)913-917
Number of pages5
JournalBehavior Research Methods
Volume47
Issue number4
DOIs
Publication statusPublished - Dec 2015

Fingerprint

Experiment
Fallacies
Inference
Likelihood Ratio
Bayes Factor
Planning
Ensemble
Replication
Misconceptions

Cite this

Wagenmakers, E-J., Verhagen, J., Ly, A., Bakker, M., Lee, M. D., Matzke, D., ... Morey, R. D. (2015). A power fallacy. Behavior Research Methods, 47(4), 913-917. https://doi.org/10.3758/s13428-014-0517-4
Wagenmakers, Eric-Jan ; Verhagen, Josine ; Ly, Alexander ; Bakker, M. ; Lee, Michael D ; Matzke, Dora ; Rouder, Jeffrey N ; Morey, Richard D. / A power fallacy. In: Behavior Research Methods. 2015 ; Vol. 47, No. 4. pp. 913-917.
@article{d1970b6b235349ea9ae46a51f1b122ea,
title = "A power fallacy",
abstract = "The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a particular outcome from a high-power experiment can be completely uninformative, whereas a particular outcome from a low-power experiment can be highly informative. Although power is useful in planning an experiment, it is less useful-and sometimes even misleading-for making inferences from observed data. To make inferences from data, we recommend the use of likelihood ratios or Bayes factors, which are the extension of likelihood ratios beyond point hypotheses. These methods of inference do not average over hypothetical replications of an experiment, but instead condition on the data that have actually been observed. In this way, likelihood ratios and Bayes factors rationally quantify the evidence that a particular data set provides for or against the null or any other hypothesis.",
author = "Eric-Jan Wagenmakers and Josine Verhagen and Alexander Ly and M. Bakker and Lee, {Michael D} and Dora Matzke and Rouder, {Jeffrey N} and Morey, {Richard D}",
year = "2015",
month = "12",
doi = "10.3758/s13428-014-0517-4",
language = "English",
volume = "47",
pages = "913--917",
journal = "Behavior Research Methods",
issn = "1554-351X",
publisher = "Springer",
number = "4",

}

Wagenmakers, E-J, Verhagen, J, Ly, A, Bakker, M, Lee, MD, Matzke, D, Rouder, JN & Morey, RD 2015, 'A power fallacy', Behavior Research Methods, vol. 47, no. 4, pp. 913-917. https://doi.org/10.3758/s13428-014-0517-4

A power fallacy. / Wagenmakers, Eric-Jan; Verhagen, Josine; Ly, Alexander; Bakker, M.; Lee, Michael D; Matzke, Dora; Rouder, Jeffrey N; Morey, Richard D.

In: Behavior Research Methods, Vol. 47, No. 4, 12.2015, p. 913-917.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - A power fallacy

AU - Wagenmakers, Eric-Jan

AU - Verhagen, Josine

AU - Ly, Alexander

AU - Bakker, M.

AU - Lee, Michael D

AU - Matzke, Dora

AU - Rouder, Jeffrey N

AU - Morey, Richard D

PY - 2015/12

Y1 - 2015/12

N2 - The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a particular outcome from a high-power experiment can be completely uninformative, whereas a particular outcome from a low-power experiment can be highly informative. Although power is useful in planning an experiment, it is less useful-and sometimes even misleading-for making inferences from observed data. To make inferences from data, we recommend the use of likelihood ratios or Bayes factors, which are the extension of likelihood ratios beyond point hypotheses. These methods of inference do not average over hypothetical replications of an experiment, but instead condition on the data that have actually been observed. In this way, likelihood ratios and Bayes factors rationally quantify the evidence that a particular data set provides for or against the null or any other hypothesis.

AB - The power fallacy refers to the misconception that what holds on average -across an ensemble of hypothetical experiments- also holds for each case individually. According to the fallacy, high-power experiments always yield more informative data than do low-power experiments. Here we expose the fallacy with concrete examples, demonstrating that a particular outcome from a high-power experiment can be completely uninformative, whereas a particular outcome from a low-power experiment can be highly informative. Although power is useful in planning an experiment, it is less useful-and sometimes even misleading-for making inferences from observed data. To make inferences from data, we recommend the use of likelihood ratios or Bayes factors, which are the extension of likelihood ratios beyond point hypotheses. These methods of inference do not average over hypothetical replications of an experiment, but instead condition on the data that have actually been observed. In this way, likelihood ratios and Bayes factors rationally quantify the evidence that a particular data set provides for or against the null or any other hypothesis.

U2 - 10.3758/s13428-014-0517-4

DO - 10.3758/s13428-014-0517-4

M3 - Article

VL - 47

SP - 913

EP - 917

JO - Behavior Research Methods

JF - Behavior Research Methods

SN - 1554-351X

IS - 4

ER -

Wagenmakers E-J, Verhagen J, Ly A, Bakker M, Lee MD, Matzke D et al. A power fallacy. Behavior Research Methods. 2015 Dec;47(4):913-917. https://doi.org/10.3758/s13428-014-0517-4