The reliability of replications: A study in computational reproductions

Nate Breznau; Eike Mark Rinke; Alexander Wuttke; Muna Adem; Jule Adriaans; Esra Akdeniz; Amalia Alvarez-Benjumea; Henrik K Andersen; Daniel Auer; Flavio Azevedo; Oke Bahnsen; Ling Bai; Dave Balzer; Paul C Bauer; Gerrit Bauer; Markus Baumann; Sharon Baute; Verena Benoit; Julian Bernauer; Carl Berning; Anna Berthold; Felix S Bethke; Thomas Biegert; Katharina Blinzler; Johannes N Blumenberg; Licia Bobzien; Andrea Bohman; Thijs Bol; Amie Bostic; Zuzanna Brzozowska; Katharina Burgdorf; Kaspar Burger; Kathrin Busch; Juan-Carlos Castillo; Nathan Chan; Pablo Christmann; Roxanne Connelly; Christian S Czymara; Elena Damian; Eline A de Rooij; Alejandro Ecker; Achim Edelmann; Christina Eder; Maureen A Eger; Simon Ellerbrock; Anna Forke; Andrea Forster; Danilo Freire; Chris Gaasendam; Konstantin Gavras; Vernon Gayle; Theresa Gessler; Timo Gnambs; Amélie Godefroidt; Max Grömping; Martin Groß; Stefan Gruber; Tobias Gummer; Andreas Hadjar; Verena Halbherr; Jan Paul Heisig; Sebastian Hellmeier; Stefanie Heyne; Magdalena Hirsch; Mikael Hjerm; Oshrat Hochman; Jan H Höffler; Andreas Hövermann; Sophia Hunger; Christian Hunkler; Nora Huth-Stöckle; Zsófia S Ignácz; Sabine Israel; Laura Jacobs; Jannes Jacobsen; Bastian Jaeger; Sebastian Jungkunz; Nils Jungmann; Jennifer Kanjana; Mathias Kauff; Salman Khan; Sayak Khatua; Manuel Kleinert; Julia Klinger; Jan-Philipp Kolb; Marta Kołczyńska; John Kuk; Katharina Kunißen; Dafina Kurti Sinatra; Alexander Langenkamp; Robin C Lee; Philipp M Lersch; David Liu; Lea-Maria Löbel; Philipp Lutscher; Matthias Mader; Joan E Madia; Natalia Malancu; Luis Maldonado; Helge Marahrens; Nicole Martin; Paul Martinez; Jochen Mayerl; Oscar J Mayorga; Robert McDonnell; Patricia McManus; Kyle McWagner; Cecil Meeusen; Daniel Meierrieks; Jonathan Mellon; Friedolin Merhout; Samuel Merk; Daniel Meyer; Leticia Micheli; Jonathan Mijs; Cristóbal Moya; Marcel Neunhoeffer; Daniel Nüst; Olav Nygård; Fabian Ochsenfeld; Gunnar Otte; Anna Pechenkina; Mark Pickup; Christopher Prosser; Louis Raes; Kevin Ralston; Miguel Ramos; Frank Reichert; Arne Roets; Jonathan Rogers; Guido Ropers; Robin Samuel; Gregor Sand; Constanza Sanhueza Petrarca; Ariela Schachter; Merlin Schaeffer; David Schieferdecker; Elmar Schlueter; Katja Schmidt; Regine Schmidt; Alexander Schmidt-Catran; Claudia Schmiedeberg; Jürgen Schneider; Martijn Schoonvelde; Julia Schulte-Cloos; Sandy Schumann; Reinhard Schunck; Julian Seuring; Henning Silber; Willem Sleegers; Nico Sonntag; Alexander Staudt; Nadia Steiber; Nils D Steiner; Sebastian Sternberg; Dieter Stiers; Dragana Stojmenovska; Nora Storz; Erich Striessnig; Anne-Kathrin Stroppe; Jordan W Suchow; Janna Teltemann; Andrey Tibajev; Brian Tung; Giacomo Vagni; Jasper Van Assche; Meta van der Linden; Jolanda van der Noll; Arno Van Hootegem; Stefan Vogtenhuber; Bogdan Voicu; Fieke Wagemans; Nadja Wehl; Hannah Werner; Brenton M Wiernik; Fabian Winter; Christof Wolf; Cary Wu; Yuki Yamada; Björn Zakula; Nan Zhang; Conrad Ziller; Stefan Zins; Tomasz Żółtak; Hung H V Nguyen

doi:10.1098/rsos.241038

The reliability of replications: A study in computational reproductions

Nate Breznau^*, Eike Mark Rinke, Alexander Wuttke, Muna Adem, Jule Adriaans, Esra Akdeniz, Amalia Alvarez-Benjumea, Henrik K Andersen, Daniel Auer, Flavio Azevedo, Oke Bahnsen, Ling Bai, Dave Balzer, Paul C Bauer, Gerrit Bauer, Markus Baumann, Sharon Baute, Verena Benoit, Julian Bernauer, Carl BerningAnna Berthold, Felix S Bethke, Thomas Biegert, Katharina Blinzler, Johannes N Blumenberg, Licia Bobzien, Andrea Bohman, Thijs Bol, Amie Bostic, Zuzanna Brzozowska, Katharina Burgdorf, Kaspar Burger, Kathrin Busch, Juan-Carlos Castillo, Nathan Chan, Pablo Christmann, Roxanne Connelly, Christian S Czymara, Elena Damian, Eline A de Rooij, Alejandro Ecker, Achim Edelmann, Christina Eder, Maureen A Eger, Simon Ellerbrock, Anna Forke, Andrea Forster, Danilo Freire, Chris Gaasendam, Konstantin Gavras, Vernon Gayle, Theresa Gessler, Timo Gnambs, Amélie Godefroidt, Max Grömping, Martin Groß, Stefan Gruber, Tobias Gummer, Andreas Hadjar, Verena Halbherr, Jan Paul Heisig, Sebastian Hellmeier, Stefanie Heyne, Magdalena Hirsch, Mikael Hjerm, Oshrat Hochman, Jan H Höffler, Andreas Hövermann, Sophia Hunger, Christian Hunkler, Nora Huth-Stöckle, Zsófia S Ignácz, Sabine Israel, Laura Jacobs, Jannes Jacobsen, Bastian Jaeger, Sebastian Jungkunz, Nils Jungmann, Jennifer Kanjana, Mathias Kauff, Salman Khan, Sayak Khatua, Manuel Kleinert, Julia Klinger, Jan-Philipp Kolb, Marta Kołczyńska, John Kuk, Katharina Kunißen, Dafina Kurti Sinatra, Alexander Langenkamp, Robin C Lee, Philipp M Lersch, David Liu, Lea-Maria Löbel, Philipp Lutscher, Matthias Mader, Joan E Madia, Natalia Malancu, Luis Maldonado, Helge Marahrens, Nicole Martin, Paul Martinez, Jochen Mayerl, Oscar J Mayorga, Robert McDonnell, Patricia McManus, Kyle McWagner, Cecil Meeusen, Daniel Meierrieks, Jonathan Mellon, Friedolin Merhout, Samuel Merk, Daniel Meyer, Leticia Micheli, Jonathan Mijs, Cristóbal Moya, Marcel Neunhoeffer, Daniel Nüst, Olav Nygård, Fabian Ochsenfeld, Gunnar Otte, Anna Pechenkina, Mark Pickup, Christopher Prosser, Louis Raes, Kevin Ralston, Miguel Ramos, Frank Reichert, Arne Roets, Jonathan Rogers, Guido Ropers, Robin Samuel, Gregor Sand, Constanza Sanhueza Petrarca, Ariela Schachter, Merlin Schaeffer, David Schieferdecker, Elmar Schlueter, Katja Schmidt, Regine Schmidt, Alexander Schmidt-Catran, Claudia Schmiedeberg, Jürgen Schneider, Martijn Schoonvelde, Julia Schulte-Cloos, Sandy Schumann, Reinhard Schunck, Julian Seuring, Henning Silber, Willem Sleegers, Nico Sonntag, Alexander Staudt, Nadia Steiber, Nils D Steiner, Sebastian Sternberg, Dieter Stiers, Dragana Stojmenovska, Nora Storz, Erich Striessnig, Anne-Kathrin Stroppe, Jordan W Suchow, Janna Teltemann, Andrey Tibajev, Brian Tung, Giacomo Vagni, Jasper Van Assche, Meta van der Linden, Jolanda van der Noll, Arno Van Hootegem, Stefan Vogtenhuber, Bogdan Voicu, Fieke Wagemans, Nadja Wehl, Hannah Werner, Brenton M Wiernik, Fabian Winter, Christof Wolf, Cary Wu, Yuki Yamada, Björn Zakula, Nan Zhang, Conrad Ziller, Stefan Zins, Tomasz Żółtak, Hung H V Nguyen

^*Corresponding author for this work

Research output: Contribution to journal › Article › Scientific › peer-review

Abstract

This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

Original language	English
Article number	241038
Number of pages	23
Journal	Royal Society Open Science
Volume	12
Issue number	3
DOIs	https://doi.org/10.1098/rsos.241038
Publication status	Published - Mar 2025

Keywords

Reliability
Replications
Computational reproduction
Social and behavioural sciences

Access to Document

10.1098/rsos.241038

The reliability of replicationsFinal published version, 1.05 MBLicence: CC BY

Supplementary material from "The Reliability of Replications: A Study in Computational Reproductions"
Raes, L. (Contributor), Figshare, 3 Feb 2025
DOI: 10.6084/m9.figshare.c.7655134.v1, https://doi.org/10.6084/m9.figshare.c.7655134.v1 and one more link, https://github.com/nbreznau/how_many_replicators (show fewer)
Dataset

Cite this

Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Bai, L., Balzer, D., Bauer, P. C., Bauer, G., Baumann, M., Baute, S., Benoit, V., Bernauer, J., ... Nguyen, H. H. V. (2025). The reliability of replications: A study in computational reproductions. Royal Society Open Science, 12(3), Article 241038. https://doi.org/10.1098/rsos.241038

@article{08f0242b1efb44659b98432ae86c8fe1,

title = "The reliability of replications: A study in computational reproductions",

abstract = "This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.",

keywords = "Reliability, Replications, Computational reproduction, Social and behavioural sciences",

author = "Nate Breznau and Rinke, {Eike Mark} and Alexander Wuttke and Muna Adem and Jule Adriaans and Esra Akdeniz and Amalia Alvarez-Benjumea and Andersen, {Henrik K} and Daniel Auer and Flavio Azevedo and Oke Bahnsen and Ling Bai and Dave Balzer and Bauer, {Paul C} and Gerrit Bauer and Markus Baumann and Sharon Baute and Verena Benoit and Julian Bernauer and Carl Berning and Anna Berthold and Bethke, {Felix S} and Thomas Biegert and Katharina Blinzler and Blumenberg, {Johannes N} and Licia Bobzien and Andrea Bohman and Thijs Bol and Amie Bostic and Zuzanna Brzozowska and Katharina Burgdorf and Kaspar Burger and Kathrin Busch and Juan-Carlos Castillo and Nathan Chan and Pablo Christmann and Roxanne Connelly and Czymara, {Christian S} and Elena Damian and {de Rooij}, {Eline A} and Alejandro Ecker and Achim Edelmann and Christina Eder and Eger, {Maureen A} and Simon Ellerbrock and Anna Forke and Andrea Forster and Danilo Freire and Chris Gaasendam and Konstantin Gavras and Vernon Gayle and Theresa Gessler and Timo Gnambs and Am{\'e}lie Godefroidt and Max Gr{\"o}mping and Martin Gro{\ss} and Stefan Gruber and Tobias Gummer and Andreas Hadjar and Verena Halbherr and Heisig, {Jan Paul} and Sebastian Hellmeier and Stefanie Heyne and Magdalena Hirsch and Mikael Hjerm and Oshrat Hochman and H{\"o}ffler, {Jan H} and Andreas H{\"o}vermann and Sophia Hunger and Christian Hunkler and Nora Huth-St{\"o}ckle and Ign{\'a}cz, {Zs{\'o}fia S} and Sabine Israel and Laura Jacobs and Jannes Jacobsen and Bastian Jaeger and Sebastian Jungkunz and Nils Jungmann and Jennifer Kanjana and Mathias Kauff and Salman Khan and Sayak Khatua and Manuel Kleinert and Julia Klinger and Jan-Philipp Kolb and Marta Ko{\l}czy{\'n}ska and John Kuk and Katharina Kuni{\ss}en and {Kurti Sinatra}, Dafina and Alexander Langenkamp and Lee, {Robin C} and Lersch, {Philipp M} and David Liu and Lea-Maria L{\"o}bel and Philipp Lutscher and Matthias Mader and Madia, {Joan E} and Natalia Malancu and Luis Maldonado and Helge Marahrens and Nicole Martin and Paul Martinez and Jochen Mayerl and Mayorga, {Oscar J} and Robert McDonnell and Patricia McManus and Kyle McWagner and Cecil Meeusen and Daniel Meierrieks and Jonathan Mellon and Friedolin Merhout and Samuel Merk and Daniel Meyer and Leticia Micheli and Jonathan Mijs and Crist{\'o}bal Moya and Marcel Neunhoeffer and Daniel N{\"u}st and Olav Nyg{\aa}rd and Fabian Ochsenfeld and Gunnar Otte and Anna Pechenkina and Mark Pickup and Christopher Prosser and Louis Raes and Kevin Ralston and Miguel Ramos and Frank Reichert and Arne Roets and Jonathan Rogers and Guido Ropers and Robin Samuel and Gregor Sand and {Sanhueza Petrarca}, Constanza and Ariela Schachter and Merlin Schaeffer and David Schieferdecker and Elmar Schlueter and Katja Schmidt and Regine Schmidt and Alexander Schmidt-Catran and Claudia Schmiedeberg and J{\"u}rgen Schneider and Martijn Schoonvelde and Julia Schulte-Cloos and Sandy Schumann and Reinhard Schunck and Julian Seuring and Henning Silber and Willem Sleegers and Nico Sonntag and Alexander Staudt and Nadia Steiber and Steiner, {Nils D} and Sebastian Sternberg and Dieter Stiers and Dragana Stojmenovska and Nora Storz and Erich Striessnig and Anne-Kathrin Stroppe and Suchow, {Jordan W} and Janna Teltemann and Andrey Tibajev and Brian Tung and Giacomo Vagni and {Van Assche}, Jasper and {van der Linden}, Meta and {van der Noll}, Jolanda and {Van Hootegem}, Arno and Stefan Vogtenhuber and Bogdan Voicu and Fieke Wagemans and Nadja Wehl and Hannah Werner and Wiernik, {Brenton M} and Fabian Winter and Christof Wolf and Cary Wu and Yuki Yamada and Bj{\"o}rn Zakula and Nan Zhang and Conrad Ziller and Stefan Zins and Tomasz {\.Z}{\'o}{\l}tak and Nguyen, {Hung H V}",

note = "{\textcopyright} 2025 The Author(s).",

year = "2025",

month = mar,

doi = "10.1098/rsos.241038",

language = "English",

volume = "12",

journal = "Royal Society Open Science",

issn = "2054-5703",

publisher = "ROYAL SOC",

number = "3",

}

Breznau, N, Rinke, EM, Wuttke, A, Adem, M, Adriaans, J, Akdeniz, E, Alvarez-Benjumea, A, Andersen, HK, Auer, D, Azevedo, F, Bahnsen, O, Bai, L, Balzer, D, Bauer, PC, Bauer, G, Baumann, M, Baute, S, Benoit, V, Bernauer, J, Berning, C, Berthold, A, Bethke, FS, Biegert, T, Blinzler, K, Blumenberg, JN, Bobzien, L, Bohman, A, Bol, T, Bostic, A, Brzozowska, Z, Burgdorf, K, Burger, K, Busch, K, Castillo, J-C, Chan, N, Christmann, P, Connelly, R, Czymara, CS, Damian, E, de Rooij, EA, Ecker, A, Edelmann, A, Eder, C, Eger, MA, Ellerbrock, S, Forke, A, Forster, A, Freire, D, Gaasendam, C, Gavras, K, Gayle, V, Gessler, T, Gnambs, T, Godefroidt, A, Grömping, M, Groß, M, Gruber, S, Gummer, T, Hadjar, A, Halbherr, V, Heisig, JP, Hellmeier, S, Heyne, S, Hirsch, M, Hjerm, M, Hochman, O, Höffler, JH, Hövermann, A, Hunger, S, Hunkler, C, Huth-Stöckle, N, Ignácz, ZS, Israel, S, Jacobs, L, Jacobsen, J, Jaeger, B, Jungkunz, S, Jungmann, N, Kanjana, J, Kauff, M, Khan, S, Khatua, S, Kleinert, M, Klinger, J, Kolb, J-P, Kołczyńska, M, Kuk, J, Kunißen, K, Kurti Sinatra, D, Langenkamp, A, Lee, RC, Lersch, PM, Liu, D, Löbel, L-M, Lutscher, P, Mader, M, Madia, JE, Malancu, N, Maldonado, L, Marahrens, H, Martin, N, Martinez, P, Mayerl, J, Mayorga, OJ, McDonnell, R, McManus, P, McWagner, K, Meeusen, C, Meierrieks, D, Mellon, J, Merhout, F, Merk, S, Meyer, D, Micheli, L, Mijs, J, Moya, C, Neunhoeffer, M, Nüst, D, Nygård, O, Ochsenfeld, F, Otte, G, Pechenkina, A, Pickup, M, Prosser, C, Raes, L, Ralston, K, Ramos, M, Reichert, F, Roets, A, Rogers, J, Ropers, G, Samuel, R, Sand, G, Sanhueza Petrarca, C, Schachter, A, Schaeffer, M, Schieferdecker, D, Schlueter, E, Schmidt, K, Schmidt, R, Schmidt-Catran, A, Schmiedeberg, C, Schneider, J, Schoonvelde, M, Schulte-Cloos, J, Schumann, S, Schunck, R, Seuring, J, Silber, H, Sleegers, W, Sonntag, N, Staudt, A, Steiber, N, Steiner, ND, Sternberg, S, Stiers, D, Stojmenovska, D, Storz, N, Striessnig, E, Stroppe, A-K, Suchow, JW, Teltemann, J, Tibajev, A, Tung, B, Vagni, G, Van Assche, J, van der Linden, M, van der Noll, J, Van Hootegem, A, Vogtenhuber, S, Voicu, B, Wagemans, F, Wehl, N, Werner, H, Wiernik, BM, Winter, F, Wolf, C, Wu, C, Yamada, Y, Zakula, B, Zhang, N, Ziller, C, Zins, S, Żółtak, T & Nguyen, HHV 2025, 'The reliability of replications: A study in computational reproductions', Royal Society Open Science, vol. 12, no. 3, 241038. https://doi.org/10.1098/rsos.241038

TY - JOUR

T1 - The reliability of replications

T2 - A study in computational reproductions

AU - Breznau, Nate

AU - Rinke, Eike Mark

AU - Wuttke, Alexander

AU - Adem, Muna

AU - Adriaans, Jule

AU - Akdeniz, Esra

AU - Alvarez-Benjumea, Amalia

AU - Andersen, Henrik K

AU - Auer, Daniel

AU - Azevedo, Flavio

AU - Bahnsen, Oke

AU - Bai, Ling

AU - Balzer, Dave

AU - Bauer, Paul C

AU - Bauer, Gerrit

AU - Baumann, Markus

AU - Baute, Sharon

AU - Benoit, Verena

AU - Bernauer, Julian

AU - Berning, Carl

AU - Berthold, Anna

AU - Bethke, Felix S

AU - Biegert, Thomas

AU - Blinzler, Katharina

AU - Blumenberg, Johannes N

AU - Bobzien, Licia

AU - Bohman, Andrea

AU - Bol, Thijs

AU - Bostic, Amie

AU - Brzozowska, Zuzanna

AU - Burgdorf, Katharina

AU - Burger, Kaspar

AU - Busch, Kathrin

AU - Castillo, Juan-Carlos

AU - Chan, Nathan

AU - Christmann, Pablo

AU - Connelly, Roxanne

AU - Czymara, Christian S

AU - Damian, Elena

AU - de Rooij, Eline A

AU - Ecker, Alejandro

AU - Edelmann, Achim

AU - Eder, Christina

AU - Eger, Maureen A

AU - Ellerbrock, Simon

AU - Forke, Anna

AU - Forster, Andrea

AU - Freire, Danilo

AU - Gaasendam, Chris

AU - Gavras, Konstantin

AU - Gayle, Vernon

AU - Gessler, Theresa

AU - Gnambs, Timo

AU - Godefroidt, Amélie

AU - Grömping, Max

AU - Groß, Martin

AU - Gruber, Stefan

AU - Gummer, Tobias

AU - Hadjar, Andreas

AU - Halbherr, Verena

AU - Heisig, Jan Paul

AU - Hellmeier, Sebastian

AU - Heyne, Stefanie

AU - Hirsch, Magdalena

AU - Hjerm, Mikael

AU - Hochman, Oshrat

AU - Höffler, Jan H

AU - Hövermann, Andreas

AU - Hunger, Sophia

AU - Hunkler, Christian

AU - Huth-Stöckle, Nora

AU - Ignácz, Zsófia S

AU - Israel, Sabine

AU - Jacobs, Laura

AU - Jacobsen, Jannes

AU - Jaeger, Bastian

AU - Jungkunz, Sebastian

AU - Jungmann, Nils

AU - Kanjana, Jennifer

AU - Kauff, Mathias

AU - Khan, Salman

AU - Khatua, Sayak

AU - Kleinert, Manuel

AU - Klinger, Julia

AU - Kolb, Jan-Philipp

AU - Kołczyńska, Marta

AU - Kuk, John

AU - Kunißen, Katharina

AU - Kurti Sinatra, Dafina

AU - Langenkamp, Alexander

AU - Lee, Robin C

AU - Lersch, Philipp M

AU - Liu, David

AU - Löbel, Lea-Maria

AU - Lutscher, Philipp

AU - Mader, Matthias

AU - Madia, Joan E

AU - Malancu, Natalia

AU - Maldonado, Luis

AU - Marahrens, Helge

AU - Martin, Nicole

AU - Martinez, Paul

AU - Mayerl, Jochen

AU - Mayorga, Oscar J

AU - McDonnell, Robert

AU - McManus, Patricia

AU - McWagner, Kyle

AU - Meeusen, Cecil

AU - Meierrieks, Daniel

AU - Mellon, Jonathan

AU - Merhout, Friedolin

AU - Merk, Samuel

AU - Meyer, Daniel

AU - Micheli, Leticia

AU - Mijs, Jonathan

AU - Moya, Cristóbal

AU - Neunhoeffer, Marcel

AU - Nüst, Daniel

AU - Nygård, Olav

AU - Ochsenfeld, Fabian

AU - Otte, Gunnar

AU - Pechenkina, Anna

AU - Pickup, Mark

AU - Prosser, Christopher

AU - Raes, Louis

AU - Ralston, Kevin

AU - Ramos, Miguel

AU - Reichert, Frank

AU - Roets, Arne

AU - Rogers, Jonathan

AU - Ropers, Guido

AU - Samuel, Robin

AU - Sand, Gregor

AU - Sanhueza Petrarca, Constanza

AU - Schachter, Ariela

AU - Schaeffer, Merlin

AU - Schieferdecker, David

AU - Schlueter, Elmar

AU - Schmidt, Katja

AU - Schmidt, Regine

AU - Schmidt-Catran, Alexander

AU - Schmiedeberg, Claudia

AU - Schneider, Jürgen

AU - Schoonvelde, Martijn

AU - Schulte-Cloos, Julia

AU - Schumann, Sandy

AU - Schunck, Reinhard

AU - Seuring, Julian

AU - Silber, Henning

AU - Sleegers, Willem

AU - Sonntag, Nico

AU - Staudt, Alexander

AU - Steiber, Nadia

AU - Steiner, Nils D

AU - Sternberg, Sebastian

AU - Stiers, Dieter

AU - Stojmenovska, Dragana

AU - Storz, Nora

AU - Striessnig, Erich

AU - Stroppe, Anne-Kathrin

AU - Suchow, Jordan W

AU - Teltemann, Janna

AU - Tibajev, Andrey

AU - Tung, Brian

AU - Vagni, Giacomo

AU - Van Assche, Jasper

AU - van der Linden, Meta

AU - van der Noll, Jolanda

AU - Van Hootegem, Arno

AU - Vogtenhuber, Stefan

AU - Voicu, Bogdan

AU - Wagemans, Fieke

AU - Wehl, Nadja

AU - Werner, Hannah

AU - Wiernik, Brenton M

AU - Winter, Fabian

AU - Wolf, Christof

AU - Wu, Cary

AU - Yamada, Yuki

AU - Zakula, Björn

AU - Zhang, Nan

AU - Ziller, Conrad

AU - Zins, Stefan

AU - Żółtak, Tomasz

AU - Nguyen, Hung H V

PY - 2025/3

Y1 - 2025/3

N2 - This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

AB - This study investigates researcher variability in computational reproduction, an activity for which it is least expected. Eighty-five independent teams attempted numerical replication of results from an original study of policy preferences and immigration. Reproduction teams were randomly grouped into a 'transparent group' receiving original study and code or 'opaque group' receiving only a method and results description and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9 and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group was reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers 'doing reproduction' would help, implying a need for better training. We also urge increased awareness of complexity in the research process and in 'push button' replications.

KW - Reliability

KW - Replications

KW - Computational reproduction

KW - Social and behavioural sciences

U2 - 10.1098/rsos.241038

DO - 10.1098/rsos.241038

M3 - Article

C2 - 40109936

SN - 2054-5703

VL - 12

JO - Royal Society Open Science

JF - Royal Society Open Science

IS - 3

M1 - 241038

ER -

The reliability of replications: A study in computational reproductions

Abstract

Keywords

Access to Document

Fingerprint

Datasets

Supplementary material from "The Reliability of Replications: A Study in Computational Reproductions"

Cite this