Investigating variation in replicability: A “Many Labs” replication project

Richard Klein; K. Ratliff; M. Vianello; R.B. Adams Jr.; S. Bahnik; M.J. Bernstein; Konrad Bocian; M.J. Brandt; Beach Brooks; Claudia Chloe Brumbaugh; Zeynep Cemalcilar; Jesse Chandler; Winnee Cheong; William E. Davis; Thierry Devos; Matthew Eisner; Natalia Frankowska; David Furrow; Elisa Maria Galliani; Fred Hasselman; Joshua A. Hicks; James F. Hovermale; S. Jane Hunt; Jeffrey R. Huntsinger; H. Ijzerman; Melissa-Sue John; Jennifer A. Joy-Gaba; Heather Barry Kappes; Lacy E. Krueger; Jamie Kurtz; Carmel A. Levitan; Robyn K. Mallett; Wendy L. Morris; Anthony J. Nelson; Kathleen Schmidt; Jeanine L. Skorinko; Robert Smith; Jason A. Nier; Grant Packard; Ronaldo Pilati; Abraham M. Rutchick; Troy G. Steiner; Justin Storbeck; Lyn M. Van Swol; Donna Thompson; Anna van 't Veer; Leigh Ann Vaughn; Marek Vranka; Aaron L. Wichman; Julie A. Woodzicka; Brian A. Nosek

doi:10.1027/1864-9335/a000178

Investigating variation in replicability: A “Many Labs” replication project

Richard Klein, K. Ratliff, M. Vianello, R.B. Adams Jr., S. Bahnik, M.J. Bernstein, Konrad Bocian, M.J. Brandt, Beach Brooks, Claudia Chloe Brumbaugh, Zeynep Cemalcilar, Jesse Chandler, Winnee Cheong, William E. Davis, Thierry Devos, Matthew Eisner, Natalia Frankowska, David Furrow, Elisa Maria Galliani, Fred HasselmanJoshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, H. Ijzerman, Melissa-Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jamie Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Troy G. Steiner, Justin Storbeck, Lyn M. Van Swol, Donna Thompson, Anna van 't Veer, Leigh Ann Vaughn, Marek Vranka, Aaron L. Wichman, Julie A. Woodzicka, Brian A. Nosek

Department of Social Psychology

Research output: Contribution to journal › Article › Scientific › peer-review

410 Citations (Scopus)

Abstract

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.
Keywords: replication, reproducibility, generalizability, cross-cultural, variation

Original language	English
Pages (from-to)	142-152
Journal	Social Psychology
Volume	45
Issue number	3
DOIs	https://doi.org/10.1027/1864-9335/a000178
Publication status	Published - 2014

Keywords

replication
reproductability
generalizability
cross-cultural
variation

Access to Document

10.1027/1864-9335/a000178

SP Ijzerman investigating SP 2014Final published version, 613 KB

Fingerprint Dive into the research topics of 'Investigating variation in replicability: A “Many Labs” replication project'. Together they form a unique fingerprint.

Cite this

Klein, R., Ratliff, K., Vianello, M., Adams Jr., R. B., Bahnik, S., Bernstein, M. J., Bocian, K., Brandt, M. J., Brooks, B., Brumbaugh, C. C., Cemalcilar, Z., Chandler, J., Cheong, W., Davis, W. E., Devos, T., Eisner, M., Frankowska, N., Furrow, D., Galliani, E. M., ... Nosek, B. A. (2014). Investigating variation in replicability: A “Many Labs” replication project. Social Psychology, 45(3), 142-152. https://doi.org/10.1027/1864-9335/a000178

Klein, Richard ; Ratliff, K. ; Vianello, M. ; Adams Jr., R.B. ; Bahnik, S. ; Bernstein, M.J. ; Bocian, Konrad ; Brandt, M.J. ; Brooks, Beach ; Brumbaugh, Claudia Chloe ; Cemalcilar, Zeynep ; Chandler, Jesse ; Cheong, Winnee ; Davis, William E. ; Devos, Thierry ; Eisner, Matthew ; Frankowska, Natalia ; Furrow, David ; Galliani, Elisa Maria ; Hasselman, Fred ; Hicks, Joshua A. ; Hovermale, James F. ; Hunt, S. Jane ; Huntsinger, Jeffrey R. ; Ijzerman, H. ; John, Melissa-Sue ; Joy-Gaba, Jennifer A. ; Kappes, Heather Barry ; Krueger, Lacy E. ; Kurtz, Jamie ; Levitan, Carmel A. ; Mallett, Robyn K. ; Morris, Wendy L. ; Nelson, Anthony J. ; Schmidt, Kathleen ; Skorinko, Jeanine L. ; Smith, Robert ; Nier, Jason A. ; Packard, Grant ; Pilati, Ronaldo ; Rutchick, Abraham M. ; Steiner, Troy G. ; Storbeck, Justin ; Van Swol, Lyn M. ; Thompson, Donna ; van 't Veer, Anna ; Vaughn, Leigh Ann ; Vranka, Marek ; Wichman, Aaron L. ; Woodzicka, Julie A. ; Nosek, Brian A. / Investigating variation in replicability: A “Many Labs” replication project. In: Social Psychology. 2014 ; Vol. 45, No. 3. pp. 142-152.

@article{a53e2ea1bea5447d8f4d8d89eb640448,

title = "Investigating variation in replicability: A “Many Labs” replication project",

abstract = "Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation",

keywords = "replication, reproductability, generalizability, cross-cultural, variation",

author = "Richard Klein and K. Ratliff and M. Vianello and {Adams Jr.}, R.B. and S. Bahnik and M.J. Bernstein and Konrad Bocian and M.J. Brandt and Beach Brooks and Brumbaugh, {Claudia Chloe} and Zeynep Cemalcilar and Jesse Chandler and Winnee Cheong and Davis, {William E.} and Thierry Devos and Matthew Eisner and Natalia Frankowska and David Furrow and Galliani, {Elisa Maria} and Fred Hasselman and Hicks, {Joshua A.} and Hovermale, {James F.} and Hunt, {S. Jane} and Huntsinger, {Jeffrey R.} and H. Ijzerman and Melissa-Sue John and Joy-Gaba, {Jennifer A.} and Kappes, {Heather Barry} and Krueger, {Lacy E.} and Jamie Kurtz and Levitan, {Carmel A.} and Mallett, {Robyn K.} and Morris, {Wendy L.} and Nelson, {Anthony J.} and Kathleen Schmidt and Skorinko, {Jeanine L.} and Robert Smith and Nier, {Jason A.} and Grant Packard and Ronaldo Pilati and Rutchick, {Abraham M.} and Steiner, {Troy G.} and Justin Storbeck and {Van Swol}, {Lyn M.} and Donna Thompson and {van 't Veer}, Anna and Vaughn, {Leigh Ann} and Marek Vranka and Wichman, {Aaron L.} and Woodzicka, {Julie A.} and Nosek, {Brian A.}",

year = "2014",

doi = "10.1027/1864-9335/a000178",

language = "English",

volume = "45",

pages = "142--152",

journal = "Social Psychology",

issn = "1864-9335",

publisher = "Hogrefe & Huber Publishers",

number = "3",

}

Klein, R, Ratliff, K, Vianello, M, Adams Jr., RB, Bahnik, S, Bernstein, MJ, Bocian, K, Brandt, MJ, Brooks, B, Brumbaugh, CC, Cemalcilar, Z, Chandler, J, Cheong, W, Davis, WE, Devos, T, Eisner, M, Frankowska, N, Furrow, D, Galliani, EM, Hasselman, F, Hicks, JA, Hovermale, JF, Hunt, SJ, Huntsinger, JR, Ijzerman, H, John, M-S, Joy-Gaba, JA, Kappes, HB, Krueger, LE, Kurtz, J, Levitan, CA, Mallett, RK, Morris, WL, Nelson, AJ, Schmidt, K, Skorinko, JL, Smith, R, Nier, JA, Packard, G, Pilati, R, Rutchick, AM, Steiner, TG, Storbeck, J, Van Swol, LM, Thompson, D, van 't Veer, A, Vaughn, LA, Vranka, M, Wichman, AL, Woodzicka, JA & Nosek, BA 2014, 'Investigating variation in replicability: A “Many Labs” replication project', Social Psychology, vol. 45, no. 3, pp. 142-152. https://doi.org/10.1027/1864-9335/a000178

Investigating variation in replicability: A “Many Labs” replication project. / Klein, Richard; Ratliff, K.; Vianello, M.; Adams Jr., R.B.; Bahnik, S.; Bernstein, M.J.; Bocian, Konrad; Brandt, M.J.; Brooks, Beach; Brumbaugh, Claudia Chloe; Cemalcilar, Zeynep; Chandler, Jesse; Cheong, Winnee; Davis, William E.; Devos, Thierry; Eisner, Matthew; Frankowska, Natalia; Furrow, David; Galliani, Elisa Maria; Hasselman, Fred; Hicks, Joshua A.; Hovermale, James F.; Hunt, S. Jane; Huntsinger, Jeffrey R.; Ijzerman, H.; John, Melissa-Sue; Joy-Gaba, Jennifer A.; Kappes, Heather Barry; Krueger, Lacy E.; Kurtz, Jamie; Levitan, Carmel A.; Mallett, Robyn K.; Morris, Wendy L.; Nelson, Anthony J.; Schmidt, Kathleen; Skorinko, Jeanine L.; Smith, Robert; Nier, Jason A.; Packard, Grant; Pilati, Ronaldo; Rutchick, Abraham M.; Steiner, Troy G.; Storbeck, Justin; Van Swol, Lyn M.; Thompson, Donna; van 't Veer, Anna; Vaughn, Leigh Ann; Vranka, Marek; Wichman, Aaron L.; Woodzicka, Julie A.; Nosek, Brian A.

In: Social Psychology, Vol. 45, No. 3, 2014, p. 142-152.

Research output: Contribution to journal › Article › Scientific › peer-review

TY - JOUR

T1 - Investigating variation in replicability: A “Many Labs” replication project

AU - Klein, Richard

AU - Ratliff, K.

AU - Vianello, M.

AU - Adams Jr., R.B.

AU - Bahnik, S.

AU - Bernstein, M.J.

AU - Bocian, Konrad

AU - Brandt, M.J.

AU - Brooks, Beach

AU - Brumbaugh, Claudia Chloe

AU - Cemalcilar, Zeynep

AU - Chandler, Jesse

AU - Cheong, Winnee

AU - Davis, William E.

AU - Devos, Thierry

AU - Eisner, Matthew

AU - Frankowska, Natalia

AU - Furrow, David

AU - Galliani, Elisa Maria

AU - Hasselman, Fred

AU - Hicks, Joshua A.

AU - Hovermale, James F.

AU - Hunt, S. Jane

AU - Huntsinger, Jeffrey R.

AU - Ijzerman, H.

AU - John, Melissa-Sue

AU - Joy-Gaba, Jennifer A.

AU - Kappes, Heather Barry

AU - Krueger, Lacy E.

AU - Kurtz, Jamie

AU - Levitan, Carmel A.

AU - Mallett, Robyn K.

AU - Morris, Wendy L.

AU - Nelson, Anthony J.

AU - Schmidt, Kathleen

AU - Skorinko, Jeanine L.

AU - Smith, Robert

AU - Nier, Jason A.

AU - Packard, Grant

AU - Pilati, Ronaldo

AU - Rutchick, Abraham M.

AU - Steiner, Troy G.

AU - Storbeck, Justin

AU - Van Swol, Lyn M.

AU - Thompson, Donna

AU - van 't Veer, Anna

AU - Vaughn, Leigh Ann

AU - Vranka, Marek

AU - Wichman, Aaron L.

AU - Woodzicka, Julie A.

AU - Nosek, Brian A.

PY - 2014

Y1 - 2014

N2 - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation

AB - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation

KW - replication

KW - reproductability

KW - generalizability

KW - cross-cultural

KW - variation

U2 - 10.1027/1864-9335/a000178

DO - 10.1027/1864-9335/a000178

M3 - Article

VL - 45

SP - 142

EP - 152

JO - Social Psychology

JF - Social Psychology

SN - 1864-9335

IS - 3

ER -