Supplementary material from "The Reliability of Replications: A Study in Computational Reproductions"

Dataset

Description

This study investigates researcher variability in computational reproduction, a place where it is least expected. Eighty-five independent teams attempted computational replication of results from an original study of policy preferences and immigration. Replication teams were randomly grouped into a ‘transparent group’ receiving original study and code, or ‘opaque group’ receiving only a method and results description, and no code. The transparent group mostly verified original results (95.7% same sign and p-value cutoff), while the opaque group had less success (89.3%). Second-decimal place exact numerical reproductions were less common (76.9% and 48.1%). Qualitative investigation of the workflows revealed many causes of error, including mistakes and procedural variations. When curating mistakes, we still find that only the transparent group were reliably successful. Our findings imply a need for transparency, but also more. Institutional checks and less subjective difficulty for researchers “doing reproduction” would help, implying a need for better training. We also suggest increased awareness of complexity in the research process and ‘push button’ replications.
Date made available3 Feb 2025
PublisherFigshare
Date of data production2018

Access Dataset

  • The reliability of replications: A study in computational reproductions

    Breznau, N., Rinke, E. M., Wuttke, A., Adem, M., Adriaans, J., Akdeniz, E., Alvarez-Benjumea, A., Andersen, H. K., Auer, D., Azevedo, F., Bahnsen, O., Bai, L., Balzer, D., Bauer, P. C., Bauer, G., Baumann, M., Baute, S., Benoit, V., Bernauer, J. & Berning, C. & 165 others, Berthold, A., Bethke, F. S., Biegert, T., Blinzler, K., Blumenberg, J. N., Bobzien, L., Bohman, A., Bol, T., Bostic, A., Brzozowska, Z., Burgdorf, K., Burger, K., Busch, K., Castillo, J.-C., Chan, N., Christmann, P., Connelly, R., Czymara, C. S., Damian, E., de Rooij, E. A., Ecker, A., Edelmann, A., Eder, C., Eger, M. A., Ellerbrock, S., Forke, A., Forster, A., Freire, D., Gaasendam, C., Gavras, K., Gayle, V., Gessler, T., Gnambs, T., Godefroidt, A., Grömping, M., Groß, M., Gruber, S., Gummer, T., Hadjar, A., Halbherr, V., Heisig, J. P., Hellmeier, S., Heyne, S., Hirsch, M., Hjerm, M., Hochman, O., Höffler, J. H., Hövermann, A., Hunger, S., Hunkler, C., Huth-Stöckle, N., Ignácz, Z. S., Israel, S., Jacobs, L., Jacobsen, J., Jaeger, B., Jungkunz, S., Jungmann, N., Kanjana, J., Kauff, M., Khan, S., Khatua, S., Kleinert, M., Klinger, J., Kolb, J.-P., Kołczyńska, M., Kuk, J., Kunißen, K., Kurti Sinatra, D., Langenkamp, A., Lee, R. C., Lersch, P. M., Liu, D., Löbel, L.-M., Lutscher, P., Mader, M., Madia, J. E., Malancu, N., Maldonado, L., Marahrens, H., Martin, N., Martinez, P., Mayerl, J., Mayorga, O. J., McDonnell, R., McManus, P., McWagner, K., Meeusen, C., Meierrieks, D., Mellon, J., Merhout, F., Merk, S., Meyer, D., Micheli, L., Mijs, J., Moya, C., Neunhoeffer, M., Nüst, D., Nygård, O., Ochsenfeld, F., Otte, G., Pechenkina, A., Pickup, M., Prosser, C., Raes, L., Ralston, K., Ramos, M., Reichert, F., Roets, A., Rogers, J., Ropers, G., Samuel, R., Sand, G., Sanhueza Petrarca, C., Schachter, A., Schaeffer, M., Schieferdecker, D., Schlueter, E., Schmidt, K., Schmidt, R., Schmidt-Catran, A., Schmiedeberg, C., Schneider, J., Schoonvelde, M., Schulte-Cloos, J., Schumann, S., Schunck, R., Seuring, J., Silber, H., Sleegers, W., Sonntag, N., Staudt, A., Steiber, N., Steiner, N. D., Sternberg, S., Stiers, D., Stojmenovska, D., Storz, N., Striessnig, E., Stroppe, A.-K., Suchow, J. W., Teltemann, J., Tibajev, A., Tung, B., Vagni, G., Van Assche, J., van der Linden, M., van der Noll, J., Van Hootegem, A., Vogtenhuber, S., Voicu, B., Wagemans, F., Wehl, N., Werner, H., Wiernik, B. M., Winter, F., Wolf, C., Wu, C., Yamada, Y., Zakula, B., Zhang, N., Ziller, C., Zins, S., Żółtak, T. & Nguyen, H. H. V., Mar 2025, In: Royal Society Open Science. 12, 3, 23 p., 241038.

    Research output: Contribution to journal › Article › Scientific › peer-review

    Open Access
    File

Cite this