Investigating variation in replicability: A “Many Labs” replication project

Richard Klein, K. Ratliff, M. Vianello, R.B. Adams Jr., S. Bahnik, M.J. Bernstein, Konrad Bocian, M.J. Brandt, Beach Brooks, Claudia Chloe Brumbaugh, Zeynep Cemalcilar, Jesse Chandler, Winnee Cheong, William E. Davis, Thierry Devos, Matthew Eisner, Natalia Frankowska, David Furrow, Elisa Maria Galliani, Fred HasselmanJoshua A. Hicks, James F. Hovermale, S. Jane Hunt, Jeffrey R. Huntsinger, H. Ijzerman, Melissa-Sue John, Jennifer A. Joy-Gaba, Heather Barry Kappes, Lacy E. Krueger, Jamie Kurtz, Carmel A. Levitan, Robyn K. Mallett, Wendy L. Morris, Anthony J. Nelson, Kathleen Schmidt, Jeanine L. Skorinko, Robert Smith, Jason A. Nier, Grant Packard, Ronaldo Pilati, Abraham M. Rutchick, Troy G. Steiner, Justin Storbeck, Lyn M. Van Swol, Donna Thompson, Anna van 't Veer, Leigh Ann Vaughn, Marek Vranka, Aaron L. Wichman, Julie A. Woodzicka, Brian A. Nosek

Research output: Contribution to journal › Article › Scientific › peer-review

359 Downloads (Pure)

Abstract

Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.
Keywords: replication, reproducibility, generalizability, cross-cultural, variation
Original languageEnglish
Pages (from-to)142-152
JournalSocial Psychology
Volume45
Issue number3
DOIs
Publication statusPublished - 2014

Fingerprint

Replication
conservatism
prejudice
currency
psychology
contact
science
Priming
Key Words
Prejudice
Currency
Justification
Reproducibility
Psychology
Tenets
Small Sample
Conservatism

Keywords

  • replication
  • reproductability
  • generalizability
  • cross-cultural
  • variation

Cite this

Klein, R., Ratliff, K., Vianello, M., Adams Jr., R. B., Bahnik, S., Bernstein, M. J., ... Nosek, B. A. (2014). Investigating variation in replicability: A “Many Labs” replication project. Social Psychology, 45(3), 142-152. https://doi.org/10.1027/1864-9335/a000178
Klein, Richard ; Ratliff, K. ; Vianello, M. ; Adams Jr., R.B. ; Bahnik, S. ; Bernstein, M.J. ; Bocian, Konrad ; Brandt, M.J. ; Brooks, Beach ; Brumbaugh, Claudia Chloe ; Cemalcilar, Zeynep ; Chandler, Jesse ; Cheong, Winnee ; Davis, William E. ; Devos, Thierry ; Eisner, Matthew ; Frankowska, Natalia ; Furrow, David ; Galliani, Elisa Maria ; Hasselman, Fred ; Hicks, Joshua A. ; Hovermale, James F. ; Hunt, S. Jane ; Huntsinger, Jeffrey R. ; Ijzerman, H. ; John, Melissa-Sue ; Joy-Gaba, Jennifer A. ; Kappes, Heather Barry ; Krueger, Lacy E. ; Kurtz, Jamie ; Levitan, Carmel A. ; Mallett, Robyn K. ; Morris, Wendy L. ; Nelson, Anthony J. ; Schmidt, Kathleen ; Skorinko, Jeanine L. ; Smith, Robert ; Nier, Jason A. ; Packard, Grant ; Pilati, Ronaldo ; Rutchick, Abraham M. ; Steiner, Troy G. ; Storbeck, Justin ; Van Swol, Lyn M. ; Thompson, Donna ; van 't Veer, Anna ; Vaughn, Leigh Ann ; Vranka, Marek ; Wichman, Aaron L. ; Woodzicka, Julie A. ; Nosek, Brian A. / Investigating variation in replicability: A “Many Labs” replication project. In: Social Psychology. 2014 ; Vol. 45, No. 3. pp. 142-152.
@article{a53e2ea1bea5447d8f4d8d89eb640448,
title = "Investigating variation in replicability: A “Many Labs” replication project",
abstract = "Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation",
keywords = "replication, reproductability, generalizability, cross-cultural, variation",
author = "Richard Klein and K. Ratliff and M. Vianello and {Adams Jr.}, R.B. and S. Bahnik and M.J. Bernstein and Konrad Bocian and M.J. Brandt and Beach Brooks and Brumbaugh, {Claudia Chloe} and Zeynep Cemalcilar and Jesse Chandler and Winnee Cheong and Davis, {William E.} and Thierry Devos and Matthew Eisner and Natalia Frankowska and David Furrow and Galliani, {Elisa Maria} and Fred Hasselman and Hicks, {Joshua A.} and Hovermale, {James F.} and Hunt, {S. Jane} and Huntsinger, {Jeffrey R.} and H. Ijzerman and Melissa-Sue John and Joy-Gaba, {Jennifer A.} and Kappes, {Heather Barry} and Krueger, {Lacy E.} and Jamie Kurtz and Levitan, {Carmel A.} and Mallett, {Robyn K.} and Morris, {Wendy L.} and Nelson, {Anthony J.} and Kathleen Schmidt and Skorinko, {Jeanine L.} and Robert Smith and Nier, {Jason A.} and Grant Packard and Ronaldo Pilati and Rutchick, {Abraham M.} and Steiner, {Troy G.} and Justin Storbeck and {Van Swol}, {Lyn M.} and Donna Thompson and {van 't Veer}, Anna and Vaughn, {Leigh Ann} and Marek Vranka and Wichman, {Aaron L.} and Woodzicka, {Julie A.} and Nosek, {Brian A.}",
year = "2014",
doi = "10.1027/1864-9335/a000178",
language = "English",
volume = "45",
pages = "142--152",
journal = "Social Psychology",
issn = "1864-9335",
publisher = "Hogrefe & Huber Publishers",
number = "3",

}

Klein, R, Ratliff, K, Vianello, M, Adams Jr., RB, Bahnik, S, Bernstein, MJ, Bocian, K, Brandt, MJ, Brooks, B, Brumbaugh, CC, Cemalcilar, Z, Chandler, J, Cheong, W, Davis, WE, Devos, T, Eisner, M, Frankowska, N, Furrow, D, Galliani, EM, Hasselman, F, Hicks, JA, Hovermale, JF, Hunt, SJ, Huntsinger, JR, Ijzerman, H, John, M-S, Joy-Gaba, JA, Kappes, HB, Krueger, LE, Kurtz, J, Levitan, CA, Mallett, RK, Morris, WL, Nelson, AJ, Schmidt, K, Skorinko, JL, Smith, R, Nier, JA, Packard, G, Pilati, R, Rutchick, AM, Steiner, TG, Storbeck, J, Van Swol, LM, Thompson, D, van 't Veer, A, Vaughn, LA, Vranka, M, Wichman, AL, Woodzicka, JA & Nosek, BA 2014, 'Investigating variation in replicability: A “Many Labs” replication project', Social Psychology, vol. 45, no. 3, pp. 142-152. https://doi.org/10.1027/1864-9335/a000178

Investigating variation in replicability: A “Many Labs” replication project. / Klein, Richard; Ratliff, K.; Vianello, M.; Adams Jr., R.B.; Bahnik, S.; Bernstein, M.J.; Bocian, Konrad; Brandt, M.J.; Brooks, Beach; Brumbaugh, Claudia Chloe; Cemalcilar, Zeynep; Chandler, Jesse; Cheong, Winnee; Davis, William E.; Devos, Thierry; Eisner, Matthew; Frankowska, Natalia; Furrow, David; Galliani, Elisa Maria; Hasselman, Fred; Hicks, Joshua A.; Hovermale, James F.; Hunt, S. Jane; Huntsinger, Jeffrey R.; Ijzerman, H.; John, Melissa-Sue; Joy-Gaba, Jennifer A.; Kappes, Heather Barry; Krueger, Lacy E.; Kurtz, Jamie; Levitan, Carmel A.; Mallett, Robyn K.; Morris, Wendy L.; Nelson, Anthony J.; Schmidt, Kathleen; Skorinko, Jeanine L.; Smith, Robert; Nier, Jason A.; Packard, Grant; Pilati, Ronaldo; Rutchick, Abraham M.; Steiner, Troy G.; Storbeck, Justin; Van Swol, Lyn M.; Thompson, Donna; van 't Veer, Anna; Vaughn, Leigh Ann; Vranka, Marek; Wichman, Aaron L.; Woodzicka, Julie A.; Nosek, Brian A.

In: Social Psychology, Vol. 45, No. 3, 2014, p. 142-152.

Research output: Contribution to journal › Article › Scientific › peer-review

TY - JOUR

T1 - Investigating variation in replicability: A “Many Labs” replication project

AU - Klein, Richard

AU - Ratliff, K.

AU - Vianello, M.

AU - Adams Jr., R.B.

AU - Bahnik, S.

AU - Bernstein, M.J.

AU - Bocian, Konrad

AU - Brandt, M.J.

AU - Brooks, Beach

AU - Brumbaugh, Claudia Chloe

AU - Cemalcilar, Zeynep

AU - Chandler, Jesse

AU - Cheong, Winnee

AU - Davis, William E.

AU - Devos, Thierry

AU - Eisner, Matthew

AU - Frankowska, Natalia

AU - Furrow, David

AU - Galliani, Elisa Maria

AU - Hasselman, Fred

AU - Hicks, Joshua A.

AU - Hovermale, James F.

AU - Hunt, S. Jane

AU - Huntsinger, Jeffrey R.

AU - Ijzerman, H.

AU - John, Melissa-Sue

AU - Joy-Gaba, Jennifer A.

AU - Kappes, Heather Barry

AU - Krueger, Lacy E.

AU - Kurtz, Jamie

AU - Levitan, Carmel A.

AU - Mallett, Robyn K.

AU - Morris, Wendy L.

AU - Nelson, Anthony J.

AU - Schmidt, Kathleen

AU - Skorinko, Jeanine L.

AU - Smith, Robert

AU - Nier, Jason A.

AU - Packard, Grant

AU - Pilati, Ronaldo

AU - Rutchick, Abraham M.

AU - Steiner, Troy G.

AU - Storbeck, Justin

AU - Van Swol, Lyn M.

AU - Thompson, Donna

AU - van 't Veer, Anna

AU - Vaughn, Leigh Ann

AU - Vranka, Marek

AU - Wichman, Aaron L.

AU - Woodzicka, Julie A.

AU - Nosek, Brian A.

PY - 2014

Y1 - 2014

N2 - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation

AB - Although replication is a central tenet of science, direct replications are rare in psychology. This research tested variation in the replicability of 13 classic and contemporary effects across 36 independent samples totaling 6,344 participants. In the aggregate, 10 effects replicated consistently. One effect – imagined contact reducing prejudice – showed weak support for replicability. And two effects – flag priming influencing conservatism and currency priming influencing system justification – did not replicate. We compared whether the conditions such as lab versus online or US versus international sample predicted effect magnitudes. By and large they did not. The results of this small sample of effects suggest that replicability is more dependent on the effect itself than on the sample and setting used to investigate the effect.Keywords: replication, reproducibility, generalizability, cross-cultural, variation

KW - replication

KW - reproductability

KW - generalizability

KW - cross-cultural

KW - variation

U2 - 10.1027/1864-9335/a000178

DO - 10.1027/1864-9335/a000178

M3 - Article

VL - 45

SP - 142

EP - 152

JO - Social Psychology

JF - Social Psychology

SN - 1864-9335

IS - 3

ER -

Klein R, Ratliff K, Vianello M, Adams Jr. RB, Bahnik S, Bernstein MJ et al. Investigating variation in replicability: A “Many Labs” replication project. Social Psychology. 2014;45(3):142-152. https://doi.org/10.1027/1864-9335/a000178

Access to Document