Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel; Barnabas Szaszi; Gustav Nilsonne; Olmo R van den Akker; Casper J Albers; Marcel  van Assen; Jojanneke A Bastiaansen; Daniel Benjamin; Udo Boehm; Rotem Botvinik-nezer; Laura F Bringmann; Niko A Busch; Emmanuel Caruyer; Andrea M Cataldo; Nelson Cowan; Andrew Delios; Noah Nn Van Dongen; Chris Donkin; Johnny B Van Doorn; Anna Dreber; Gilles Dutilh; Gary F Egan; Morton Ann Gernsbacher; Rink Hoekstra; Sabine Hoffmann; Felix Holzmeister; Juergen Huber; Magnus Johannesson; Kai J Jonas; Alexander T Kindel; Michael Kirchler; Yoram K Kunkels; D Stephen Lindsay; Jean-francois Mangin; Dora Matzke; Marcus R Munafò; Ben R Newell; Brian A Nosek; Russell A Poldrack; Don Van Ravenzwaaij; Jörg Rieskamp; Matthew J Salganik; Alexandra Sarafoglou; Tom Schonberg; Martin Schweinsberg; David Shanks; Raphael Silberzahn; Daniel J Simons; Barbara A Spellman; Samuel St-jean; Jeffrey J Starns; Eric Luis Uhlmann; Jelte Wicherts; Eric-jan Wagenmakers

doi:10.7554/eLife.72185

Consensus-based guidance for conducting and reporting multi-analyst studies

Balazs Aczel^*, Barnabas Szaszi^*, Gustav Nilsonne, Olmo R van den Akker, Casper J Albers, Marcel van Assen, Jojanneke A Bastiaansen, Daniel Benjamin, Udo Boehm, Rotem Botvinik-nezer, Laura F Bringmann, Niko A Busch, Emmanuel Caruyer, Andrea M Cataldo, Nelson Cowan, Andrew Delios, Noah Nn Van Dongen, Chris Donkin, Johnny B Van Doorn, Anna DreberGilles Dutilh, Gary F Egan, Morton Ann Gernsbacher, Rink Hoekstra, Sabine Hoffmann, Felix Holzmeister, Juergen Huber, Magnus Johannesson, Kai J Jonas, Alexander T Kindel, Michael Kirchler, Yoram K Kunkels, D Stephen Lindsay, Jean-francois Mangin, Dora Matzke, Marcus R Munafò, Ben R Newell, Brian A Nosek, Russell A Poldrack, Don Van Ravenzwaaij, Jörg Rieskamp, Matthew J Salganik, Alexandra Sarafoglou, Tom Schonberg, Martin Schweinsberg, David Shanks, Raphael Silberzahn, Daniel J Simons, Barbara A Spellman, Samuel St-jean, Jeffrey J Starns, Eric Luis Uhlmann, Jelte Wicherts, Eric-jan Wagenmakers

^*Corresponding author for this work

Methodology and Statistics

Research output: Contribution to journal › Article › Scientific › peer-review

26 Citations (Scopus)

64 Downloads (Pure)

Abstract

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

Original language	English
Article number	e72185
Number of pages	13
Journal	Elife
Volume	10
DOIs	https://doi.org/10.7554/eLife.72185
Publication status	Published - 2021

Keywords

DELPHI
None
REPRODUCIBILITY
analytical variability
expert consensus
metascience
multi-analyst
science forum
statistical practice

Access to Document

10.7554/eLife.72185Licence: CC BY

MTO vd Akker consensus-based guidance eLife 2021Final published version, 468 KBLicence: CC BY

Cite this

Aczel, B., Szaszi, B., Nilsonne, G., van den Akker, O. R., Albers, C. J., van Assen, M., Bastiaansen, J. A., Benjamin, D., Boehm, U., Botvinik-nezer, R., Bringmann, L. F., Busch, N. A., Caruyer, E., Cataldo, A. M., Cowan, N., Delios, A., Van Dongen, N. N., Donkin, C., Van Doorn, J. B., ... Wagenmakers, E. (2021). Consensus-based guidance for conducting and reporting multi-analyst studies. Elife, 10, Article e72185. https://doi.org/10.7554/eLife.72185

@article{063dbf16e47e4a3cba06a39382fa89bf,

title = "Consensus-based guidance for conducting and reporting multi-analyst studies",

abstract = "Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.",

keywords = "DELPHI, None, REPRODUCIBILITY, analytical variability, expert consensus, metascience, multi-analyst, science forum, statistical practice",

author = "Balazs Aczel and Barnabas Szaszi and Gustav Nilsonne and {van den Akker}, {Olmo R} and Albers, {Casper J} and {van Assen}, Marcel and Bastiaansen, {Jojanneke A} and Daniel Benjamin and Udo Boehm and Rotem Botvinik-nezer and Bringmann, {Laura F} and Busch, {Niko A} and Emmanuel Caruyer and Cataldo, {Andrea M} and Nelson Cowan and Andrew Delios and {Van Dongen}, {Noah Nn} and Chris Donkin and {Van Doorn}, {Johnny B} and Anna Dreber and Gilles Dutilh and Egan, {Gary F} and Gernsbacher, {Morton Ann} and Rink Hoekstra and Sabine Hoffmann and Felix Holzmeister and Juergen Huber and Magnus Johannesson and Jonas, {Kai J} and Kindel, {Alexander T} and Michael Kirchler and Kunkels, {Yoram K} and Lindsay, {D Stephen} and Jean-francois Mangin and Dora Matzke and Munaf{\`o}, {Marcus R} and Newell, {Ben R} and Nosek, {Brian A} and Poldrack, {Russell A} and {Van Ravenzwaaij}, Don and J{\"o}rg Rieskamp and Salganik, {Matthew J} and Alexandra Sarafoglou and Tom Schonberg and Martin Schweinsberg and David Shanks and Raphael Silberzahn and Simons, {Daniel J} and Spellman, {Barbara A} and Samuel St-jean and Starns, {Jeffrey J} and Uhlmann, {Eric Luis} and Jelte Wicherts and Eric-jan Wagenmakers",

note = "Funding Netherlands Organisation for Scientific Research (406-17-568) Alexandra Sarafoglou Natural Sciences and Engineering Research Council of Canada (BP-546283-2020) Samuel St-Jean Fonds de Recherche du Qu{\'e}bec - Nature et Technologies (290978) Samuel St-Jean European Research Council (726361) Jelte Wicherts Olmo R van den Akker European Research Council (681466) Yoram K Kunkels VIDI fellowship organisation (016.Vidi.188.001) Don van Ravenzwaaij VENI fellowship grant (Veni 191G.037) Laura F Bringmann National Science Foundation (1760052) Matthew J Salganik Weizmann Institute of Science (Israel National Postdoctoral Award Program for Advancing Women in Science) Rotem Botvinik-Nezer John Templeton Foundation, Templeton World Charity Foundation, Templeton Religion Trust, and Arnold Ventures Brian A Nosek Institut Europ{\'e}en d'Administration des Affaires Eric Luis Uhlmann European Research Council (640638) Noah NN van Dongen ",

year = "2021",

doi = "10.7554/eLife.72185",

language = "English",

volume = "10",

journal = "Elife",

issn = "2050-084X",

publisher = "eLife Sciences Publications Ltd",

}

Aczel, B, Szaszi, B, Nilsonne, G, van den Akker, OR, Albers, CJ, van Assen, M, Bastiaansen, JA, Benjamin, D, Boehm, U, Botvinik-nezer, R, Bringmann, LF, Busch, NA, Caruyer, E, Cataldo, AM, Cowan, N, Delios, A, Van Dongen, NN, Donkin, C, Van Doorn, JB, Dreber, A, Dutilh, G, Egan, GF, Gernsbacher, MA, Hoekstra, R, Hoffmann, S, Holzmeister, F, Huber, J, Johannesson, M, Jonas, KJ, Kindel, AT, Kirchler, M, Kunkels, YK, Lindsay, DS, Mangin, J, Matzke, D, Munafò, MR, Newell, BR, Nosek, BA, Poldrack, RA, Van Ravenzwaaij, D, Rieskamp, J, Salganik, MJ, Sarafoglou, A, Schonberg, T, Schweinsberg, M, Shanks, D, Silberzahn, R, Simons, DJ, Spellman, BA, St-jean, S, Starns, JJ, Uhlmann, EL, Wicherts, J & Wagenmakers, E 2021, 'Consensus-based guidance for conducting and reporting multi-analyst studies', Elife, vol. 10, e72185. https://doi.org/10.7554/eLife.72185

TY - JOUR

T1 - Consensus-based guidance for conducting and reporting multi-analyst studies

AU - Aczel, Balazs

AU - Szaszi, Barnabas

AU - Nilsonne, Gustav

AU - van den Akker, Olmo R

AU - Albers, Casper J

AU - van Assen, Marcel

AU - Bastiaansen, Jojanneke A

AU - Benjamin, Daniel

AU - Boehm, Udo

AU - Botvinik-nezer, Rotem

AU - Bringmann, Laura F

AU - Busch, Niko A

AU - Caruyer, Emmanuel

AU - Cataldo, Andrea M

AU - Cowan, Nelson

AU - Delios, Andrew

AU - Van Dongen, Noah Nn

AU - Donkin, Chris

AU - Van Doorn, Johnny B

AU - Dreber, Anna

AU - Dutilh, Gilles

AU - Egan, Gary F

AU - Gernsbacher, Morton Ann

AU - Hoekstra, Rink

AU - Hoffmann, Sabine

AU - Holzmeister, Felix

AU - Huber, Juergen

AU - Johannesson, Magnus

AU - Jonas, Kai J

AU - Kindel, Alexander T

AU - Kirchler, Michael

AU - Kunkels, Yoram K

AU - Lindsay, D Stephen

AU - Mangin, Jean-francois

AU - Matzke, Dora

AU - Munafò, Marcus R

AU - Newell, Ben R

AU - Nosek, Brian A

AU - Poldrack, Russell A

AU - Van Ravenzwaaij, Don

AU - Rieskamp, Jörg

AU - Salganik, Matthew J

AU - Sarafoglou, Alexandra

AU - Schonberg, Tom

AU - Schweinsberg, Martin

AU - Shanks, David

AU - Silberzahn, Raphael

AU - Simons, Daniel J

AU - Spellman, Barbara A

AU - St-jean, Samuel

AU - Starns, Jeffrey J

AU - Uhlmann, Eric Luis

AU - Wicherts, Jelte

AU - Wagenmakers, Eric-jan

N1 - Funding Netherlands Organisation for Scientific Research (406-17-568) Alexandra Sarafoglou Natural Sciences and Engineering Research Council of Canada (BP-546283-2020) Samuel St-Jean Fonds de Recherche du Québec - Nature et Technologies (290978) Samuel St-Jean European Research Council (726361) Jelte Wicherts Olmo R van den Akker European Research Council (681466) Yoram K Kunkels VIDI fellowship organisation (016.Vidi.188.001) Don van Ravenzwaaij VENI fellowship grant (Veni 191G.037) Laura F Bringmann National Science Foundation (1760052) Matthew J Salganik Weizmann Institute of Science (Israel National Postdoctoral Award Program for Advancing Women in Science) Rotem Botvinik-Nezer John Templeton Foundation, Templeton World Charity Foundation, Templeton Religion Trust, and Arnold Ventures Brian A Nosek Institut Européen d'Administration des Affaires Eric Luis Uhlmann European Research Council (640638) Noah NN van Dongen

PY - 2021

Y1 - 2021

N2 - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

AB - Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

KW - DELPHI

KW - None

KW - REPRODUCIBILITY

KW - analytical variability

KW - expert consensus

KW - metascience

KW - multi-analyst

KW - science forum

KW - statistical practice

UR - http://www.scopus.com/inward/record.url?scp=85120181656&partnerID=8YFLogxK

U2 - 10.7554/eLife.72185

DO - 10.7554/eLife.72185

M3 - Article

SN - 2050-084X

VL - 10

JO - Elife

JF - Elife

M1 - e72185

ER -

Consensus-based guidance for conducting and reporting multi-analyst studies

Abstract

Keywords

Access to Document

Other files and links

Fingerprint

Cite this