Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Charles R. Ebersole; Olivia E. Atherton; Aimee L. Belanger; Hayley M. Skulborstad; Jill M. Allen; Jonathan B. Banks; Erica Baranski; Michael J. Bernstein; Diane B. V. Bonfiglio; Leanne Boucher; Elizabeth R. Brown; Nancy I. Budiman; Athena H. Cairo; Colin A. Capaldi; Christopher R. Chartier; J.M.H. Chung; David C. Cicero; Jennifer A. Coleman; John G. Conway; William E. Davis; Thierry Devos; Melody M. Fletcher; Komi German; Jon E. Grahe; Anthony D. Hermann; Joshua A. Hicks; Nathan Honeycutt; Brandon Humphrey; Matthew Janus; David J. Johnson; Jennifer A. Joy-Gaba; Hannah Juzeler; Ashley Keres; Diana Kinney; Jacqeline Kirshenbaum; Richard A. Klein; Richard E. Lucas; Christopher J. N. Lustgraaf; Daniel Martin; Madhavi Menon; Mitchell Metzger; Jaclyn M. Moloney; Patrick J. Morse; Radmila Prislin; Timothy Razza; Daniel E. Re; Nicholas O. Rule; Tdonald F. Sacco; Kyle Sauerberger; Emily Shrider; Megan Shultz; Courtney Siemsen; Karin Sobocko; R. Weylin Sternglanz; Amy Summerville; Konstantin O. Tskhay; Zack van Allen; Leigh Ann Vaughn; Ryan J. Walker; Ashley Weinberg; John Paul Wilson; James H. Wirth; Jessica Wortman; Brian A. Nosek

doi:10.1016/j.jesp.2015.10.012

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Charles R. Ebersole^*, Olivia E. Atherton, Aimee L. Belanger, Hayley M. Skulborstad, Jill M. Allen, Jonathan B. Banks, Erica Baranski, Michael J. Bernstein, Diane B. V. Bonfiglio, Leanne Boucher, Elizabeth R. Brown, Nancy I. Budiman, Athena H. Cairo, Colin A. Capaldi, Christopher R. Chartier, J.M.H. Chung, David C. Cicero, Jennifer A. Coleman, John G. Conway, William E. DavisThierry Devos, Melody M. Fletcher, Komi German, Jon E. Grahe, Anthony D. Hermann, Joshua A. Hicks, Nathan Honeycutt, Brandon Humphrey, Matthew Janus, David J. Johnson, Jennifer A. Joy-Gaba, Hannah Juzeler, Ashley Keres, Diana Kinney, Jacqeline Kirshenbaum, Richard A. Klein, Richard E. Lucas, Christopher J. N. Lustgraaf, Daniel Martin, Madhavi Menon, Mitchell Metzger, Jaclyn M. Moloney, Patrick J. Morse, Radmila Prislin, Timothy Razza, Daniel E. Re, Nicholas O. Rule, Tdonald F. Sacco, Kyle Sauerberger, Emily Shrider, Megan Shultz, Courtney Siemsen, Karin Sobocko, R. Weylin Sternglanz, Amy Summerville, Konstantin O. Tskhay, Zack van Allen, Leigh Ann Vaughn, Ryan J. Walker, Ashley Weinberg, John Paul Wilson, James H. Wirth, Jessica Wortman, Brian A. Nosek

^*Corresponding author for this work

Developmental Psychology

Research output: Contribution to journal › Article › Scientific › peer-review

244 Citations (Scopus)

Abstract

The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. (C) 2015 Elsevier Inc. All rights reserved.

Original language	English
Pages (from-to)	68-82
Journal	Journal of Experimental Social Psychology
Volume	67
DOIs	https://doi.org/10.1016/j.jesp.2015.10.012
Publication status	Published - Nov 2016

Keywords

Social psychology
Cognitive psychology
Replication
Participant pool
Individual differences
Sampling effects
Situational effects
SELF-ESTEEM
SUBJECT POOL
PERSONALITY
COGNITION
NEED
TIME
INTERFERENCE
METAANALYSIS
INFORMATION
METAPHORS

Access to Document

10.1016/j.jesp.2015.10.012

Cite this

Ebersole, C. R., Atherton, O. E., Belanger, A. L., Skulborstad, H. M., Allen, J. M., Banks, J. B., Baranski, E., Bernstein, M. J., Bonfiglio, D. B. V., Boucher, L., Brown, E. R., Budiman, N. I., Cairo, A. H., Capaldi, C. A., Chartier, C. R., Chung, J. M. H., Cicero, D. C., Coleman, J. A., Conway, J. G., ... Nosek, B. A. (2016). Many Labs 3: Evaluating participant pool quality across the academic semester via replication. Journal of Experimental Social Psychology, 67, 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

@article{1933bef5b2bc4f3ab657582a85d7217c,

title = "Many Labs 3: Evaluating participant pool quality across the academic semester via replication",

abstract = "The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. (C) 2015 Elsevier Inc. All rights reserved.",

keywords = "Social psychology, Cognitive psychology, Replication, Participant pool, Individual differences, Sampling effects, Situational effects, SELF-ESTEEM, SUBJECT POOL, PERSONALITY, COGNITION, NEED, TIME, INTERFERENCE, METAANALYSIS, INFORMATION, METAPHORS",

author = "Ebersole, {Charles R.} and Atherton, {Olivia E.} and Belanger, {Aimee L.} and Skulborstad, {Hayley M.} and Allen, {Jill M.} and Banks, {Jonathan B.} and Erica Baranski and Bernstein, {Michael J.} and Bonfiglio, {Diane B. V.} and Leanne Boucher and Brown, {Elizabeth R.} and Budiman, {Nancy I.} and Cairo, {Athena H.} and Capaldi, {Colin A.} and Chartier, {Christopher R.} and J.M.H. Chung and Cicero, {David C.} and Coleman, {Jennifer A.} and Conway, {John G.} and Davis, {William E.} and Thierry Devos and Fletcher, {Melody M.} and Komi German and Grahe, {Jon E.} and Hermann, {Anthony D.} and Hicks, {Joshua A.} and Nathan Honeycutt and Brandon Humphrey and Matthew Janus and Johnson, {David J.} and Joy-Gaba, {Jennifer A.} and Hannah Juzeler and Ashley Keres and Diana Kinney and Jacqeline Kirshenbaum and Klein, {Richard A.} and Lucas, {Richard E.} and Lustgraaf, {Christopher J. N.} and Daniel Martin and Madhavi Menon and Mitchell Metzger and Moloney, {Jaclyn M.} and Morse, {Patrick J.} and Radmila Prislin and Timothy Razza and Re, {Daniel E.} and Rule, {Nicholas O.} and Sacco, {Tdonald F.} and Kyle Sauerberger and Emily Shrider and Megan Shultz and Courtney Siemsen and Karin Sobocko and Sternglanz, {R. Weylin} and Amy Summerville and Tskhay, {Konstantin O.} and {van Allen}, Zack and Vaughn, {Leigh Ann} and Walker, {Ryan J.} and Ashley Weinberg and Wilson, {John Paul} and Wirth, {James H.} and Jessica Wortman and Nosek, {Brian A.}",

year = "2016",

month = nov,

doi = "10.1016/j.jesp.2015.10.012",

language = "English",

volume = "67",

pages = "68--82",

journal = "Journal of Experimental Social Psychology",

issn = "0022-1031",

publisher = "ACADEMIC PRESS INC ELSEVIER SCIENCE",

}

Ebersole, CR, Atherton, OE, Belanger, AL, Skulborstad, HM, Allen, JM, Banks, JB, Baranski, E, Bernstein, MJ, Bonfiglio, DBV, Boucher, L, Brown, ER, Budiman, NI, Cairo, AH, Capaldi, CA, Chartier, CR, Chung, JMH, Cicero, DC, Coleman, JA, Conway, JG, Davis, WE, Devos, T, Fletcher, MM, German, K, Grahe, JE, Hermann, AD, Hicks, JA, Honeycutt, N, Humphrey, B, Janus, M, Johnson, DJ, Joy-Gaba, JA, Juzeler, H, Keres, A, Kinney, D, Kirshenbaum, J, Klein, RA, Lucas, RE, Lustgraaf, CJN, Martin, D, Menon, M, Metzger, M, Moloney, JM, Morse, PJ, Prislin, R, Razza, T, Re, DE, Rule, NO, Sacco, TF, Sauerberger, K, Shrider, E, Shultz, M, Siemsen, C, Sobocko, K, Sternglanz, RW, Summerville, A, Tskhay, KO, van Allen, Z, Vaughn, LA, Walker, RJ, Weinberg, A, Wilson, JP, Wirth, JH, Wortman, J & Nosek, BA 2016, 'Many Labs 3: Evaluating participant pool quality across the academic semester via replication', Journal of Experimental Social Psychology, vol. 67, pp. 68-82. https://doi.org/10.1016/j.jesp.2015.10.012

TY - JOUR

T1 - Many Labs 3

T2 - Evaluating participant pool quality across the academic semester via replication

AU - Ebersole, Charles R.

AU - Atherton, Olivia E.

AU - Belanger, Aimee L.

AU - Skulborstad, Hayley M.

AU - Allen, Jill M.

AU - Banks, Jonathan B.

AU - Baranski, Erica

AU - Bernstein, Michael J.

AU - Bonfiglio, Diane B. V.

AU - Boucher, Leanne

AU - Brown, Elizabeth R.

AU - Budiman, Nancy I.

AU - Cairo, Athena H.

AU - Capaldi, Colin A.

AU - Chartier, Christopher R.

AU - Chung, J.M.H.

AU - Cicero, David C.

AU - Coleman, Jennifer A.

AU - Conway, John G.

AU - Davis, William E.

AU - Devos, Thierry

AU - Fletcher, Melody M.

AU - German, Komi

AU - Grahe, Jon E.

AU - Hermann, Anthony D.

AU - Hicks, Joshua A.

AU - Honeycutt, Nathan

AU - Humphrey, Brandon

AU - Janus, Matthew

AU - Johnson, David J.

AU - Joy-Gaba, Jennifer A.

AU - Juzeler, Hannah

AU - Keres, Ashley

AU - Kinney, Diana

AU - Kirshenbaum, Jacqeline

AU - Klein, Richard A.

AU - Lucas, Richard E.

AU - Lustgraaf, Christopher J. N.

AU - Martin, Daniel

AU - Menon, Madhavi

AU - Metzger, Mitchell

AU - Moloney, Jaclyn M.

AU - Morse, Patrick J.

AU - Prislin, Radmila

AU - Razza, Timothy

AU - Re, Daniel E.

AU - Rule, Nicholas O.

AU - Sacco, Tdonald F.

AU - Sauerberger, Kyle

AU - Shrider, Emily

AU - Shultz, Megan

AU - Siemsen, Courtney

AU - Sobocko, Karin

AU - Sternglanz, R. Weylin

AU - Summerville, Amy

AU - Tskhay, Konstantin O.

AU - van Allen, Zack

AU - Vaughn, Leigh Ann

AU - Walker, Ryan J.

AU - Weinberg, Ashley

AU - Wilson, John Paul

AU - Wirth, James H.

AU - Wortman, Jessica

AU - Nosek, Brian A.

PY - 2016/11

Y1 - 2016/11

N2 - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. (C) 2015 Elsevier Inc. All rights reserved.

AB - The university participant pool is a key resource for behavioral research, and data quality is believed to vary over the course of the academic semester. This crowdsourced project examined time of semester variation in 10 known effects, 10 individual differences, and 3 data quality indicators over the course of the academic semester in 20 participant pools (N = 2696) and with an online sample (N = 737). Weak time of semester effects were observed on data quality indicators, participant sex, and a few individual differences conscientiousness, mood, and stress. However, there was little evidence for time of semester qualifying experimental or correlational effects. The generality of this evidence is unknown because only a subset of the tested effects demonstrated evidence for the original result in the whole sample. Mean characteristics of pool samples change slightly during the semester, but these data suggest that those changes are mostly irrelevant for detecting effects. (C) 2015 Elsevier Inc. All rights reserved.

KW - Social psychology

KW - Cognitive psychology

KW - Replication

KW - Participant pool

KW - Individual differences

KW - Sampling effects

KW - Situational effects

KW - SELF-ESTEEM

KW - SUBJECT POOL

KW - PERSONALITY

KW - COGNITION

KW - NEED

KW - TIME

KW - INTERFERENCE

KW - METAANALYSIS

KW - INFORMATION

KW - METAPHORS

U2 - 10.1016/j.jesp.2015.10.012

DO - 10.1016/j.jesp.2015.10.012

M3 - Article

SN - 0022-1031

VL - 67

SP - 68

EP - 82

JO - Journal of Experimental Social Psychology

JF - Journal of Experimental Social Psychology

ER -

Many Labs 3: Evaluating participant pool quality across the academic semester via replication

Abstract

Keywords

Access to Document

Fingerprint

Cite this