Statistical reporting errors and collaboration on statistical analyses in psychological science

C.L.S. Veldkamp, M.B. Nuijten, L. Dominguez Alvarez, M.A.L.M. van Assen, J.M. Wicherts

Research output: Contribution to journalArticleScientificpeer-review

49 Downloads (Pure)

Abstract

Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.
Original languageEnglish
Article numbere114876
JournalPLoS ONE
Volume9
Issue number12
DOIs
Publication statusPublished - 2014

Fingerprint

Statistical methods
Information Dissemination
Statistics
Practice Guidelines
Cross-Sectional Studies
Aviation

Cite this

@article{1664550a5b6f489e9abf32f94651fee1,
title = "Statistical reporting errors and collaboration on statistical analyses in psychological science",
abstract = "Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6{\%} of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63{\%} of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20{\%} of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10{\%}. Co-piloting was not found to be associated with reporting errors.",
author = "C.L.S. Veldkamp and M.B. Nuijten and {Dominguez Alvarez}, L. and {van Assen}, M.A.L.M. and J.M. Wicherts",
year = "2014",
doi = "10.1371/journal.pone.0114876",
language = "English",
volume = "9",
journal = "PLoS ONE",
issn = "1932-6203",
publisher = "PUBLIC LIBRARY SCIENCE",
number = "12",

}

Statistical reporting errors and collaboration on statistical analyses in psychological science. / Veldkamp, C.L.S.; Nuijten, M.B.; Dominguez Alvarez, L.; van Assen, M.A.L.M.; Wicherts, J.M.

In: PLoS ONE, Vol. 9, No. 12, e114876, 2014.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Statistical reporting errors and collaboration on statistical analyses in psychological science

AU - Veldkamp, C.L.S.

AU - Nuijten, M.B.

AU - Dominguez Alvarez, L.

AU - van Assen, M.A.L.M.

AU - Wicherts, J.M.

PY - 2014

Y1 - 2014

N2 - Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

AB - Statistical analysis is error prone. A best practice for researchers using statistics would therefore be to share data among co-authors, allowing double-checking of executed tasks just as co-pilots do in aviation. To document the extent to which this ‘co-piloting’ currently occurs in psychology, we surveyed the authors of 697 articles published in six top psychology journals and asked them whether they had collaborated on four aspects of analyzing data and reporting results, and whether the described data had been shared between the authors. We acquired responses for 49.6% of the articles and found that co-piloting on statistical analysis and reporting results is quite uncommon among psychologists, while data sharing among co-authors seems reasonably but not completely standard. We then used an automated procedure to study the prevalence of statistical reporting errors in the articles in our sample and examined the relationship between reporting errors and co-piloting. Overall, 63% of the articles contained at least one p-value that was inconsistent with the reported test statistic and the accompanying degrees of freedom, and 20% of the articles contained at least one p-value that was inconsistent to such a degree that it may have affected decisions about statistical significance. Overall, the probability that a given p-value was inconsistent was over 10%. Co-piloting was not found to be associated with reporting errors.

U2 - 10.1371/journal.pone.0114876

DO - 10.1371/journal.pone.0114876

M3 - Article

VL - 9

JO - PLoS ONE

JF - PLoS ONE

SN - 1932-6203

IS - 12

M1 - e114876

ER -