VALID: A checklist-based approach for improving validity in psychological research

Susanne Kerschbaumer; Martin Voracek; Balazs Aczel; Samantha F. Anderson; Brandon M. Booth; Erin M. Buchanan; Rickard Carlsson; Daniel W. Heck; Anu Pauliina Hiekkaranta; Rink Hoekstra; Julian D. Karch; Ginette Lafit; Zhicheng Lin; Siwei Liu; David P. Mackinnon; Emma L. Mcgorray; David Moreau; Marietta Papadatou-Pastou; Helena Paterson; Robert A. Perera; Daniel J. Schad; David K. Sewell; Moin Syed; Louis Tay; Jorge N. Tendeiro; Michael D. Toland; Wolf Vanpaemel; Don van Ravenzwaaij; Lieke Voncken; Ulrich S. Tran

doi:10.1177/25152459241306432

VALID: A checklist-based approach for improving validity in psychological research

Susanne Kerschbaumer^*, Martin Voracek, Balazs Aczel, Samantha F. Anderson, Brandon M. Booth, Erin M. Buchanan, Rickard Carlsson, Daniel W. Heck, Anu Pauliina Hiekkaranta, Rink Hoekstra, Julian D. Karch, Ginette Lafit, Zhicheng Lin, Siwei Liu, David P. Mackinnon, Emma L. Mcgorray, David Moreau, Marietta Papadatou-Pastou, Helena Paterson, Robert A. PereraDaniel J. Schad, David K. Sewell, Moin Syed, Louis Tay, Jorge N. Tendeiro, Michael D. Toland, Wolf Vanpaemel, Don van Ravenzwaaij, Lieke Voncken, Ulrich S. Tran

^*Corresponding author for this work

Methodology and Statistics

Research output: Contribution to journal › Article › Scientific › peer-review

Abstract

In response to the replication and confidence crisis across various empirical disciplines, ensuring the validity of research has gained attention. High validity is crucial for obtaining replicable and robust study outcomes when both exploring new questions and replicating previous findings. In this study, we aimed to address this issue by developing a comprehensive checklist to assist researchers in enhancing and monitoring the validity of their research. After systematically analyzing previous findings on validity, a comprehensive list of potential checklist items was compiled. Over the course of three rounds, more than 30 interdisciplinary and psychological-science experts participated in a Delphi study. Experts rated items on their importance and were given the opportunity to propose novel items as well as improve existing ones. This process resulted in a final set of 91 items, organized according to common stages of a research project. The VALID checklist is accessible online (https://www.validchecklist.com/) and provides researchers with an adaptable, versatile tool to monitor and improve the validity of their research and to suit their specific needs. By focusing on adaptiveness during its development, VALID encompasses 331 unique checklist versions, making it a one-stop solution suitable for a wide range of projects, designs, and requirements.

Original language	English
Article number	25152459241306432
Number of pages	16
Journal	Advances in Methods and Practices in Psychological Science
Volume	8
Issue number	1
Early online date	20 Feb 2025
DOIs	https://doi.org/10.1177/25152459241306432
Publication status	Published - Mar 2025

Keywords

Delphi study
Method reform
Open data
Open materials
Open practices
Preregistered
Study quality appraisal
Tailored online checklist
Validity

Access to Document

10.1177/25152459241306432Licence: CC BY-NC

VALID A Checklist-Based ApproachFinal published version, 2.5 MBLicence: CC BY-NC

Cite this

Kerschbaumer, S., Voracek, M., Aczel, B., Anderson, S. F., Booth, B. M., Buchanan, E. M., Carlsson, R., Heck, D. W., Hiekkaranta, A. P., Hoekstra, R., Karch, J. D., Lafit, G., Lin, Z., Liu, S., Mackinnon, D. P., Mcgorray, E. L., Moreau, D., Papadatou-Pastou, M., Paterson, H., ... Tran, U. S. (2025). VALID: A checklist-based approach for improving validity in psychological research. Advances in Methods and Practices in Psychological Science, 8(1), Article 25152459241306432. https://doi.org/10.1177/25152459241306432

@article{1c58ee0b9e4c4a469f29c0ee8033b5f3,

title = "VALID: A checklist-based approach for improving validity in psychological research",

abstract = "In response to the replication and confidence crisis across various empirical disciplines, ensuring the validity of research has gained attention. High validity is crucial for obtaining replicable and robust study outcomes when both exploring new questions and replicating previous findings. In this study, we aimed to address this issue by developing a comprehensive checklist to assist researchers in enhancing and monitoring the validity of their research. After systematically analyzing previous findings on validity, a comprehensive list of potential checklist items was compiled. Over the course of three rounds, more than 30 interdisciplinary and psychological-science experts participated in a Delphi study. Experts rated items on their importance and were given the opportunity to propose novel items as well as improve existing ones. This process resulted in a final set of 91 items, organized according to common stages of a research project. The VALID checklist is accessible online (https://www.validchecklist.com/) and provides researchers with an adaptable, versatile tool to monitor and improve the validity of their research and to suit their specific needs. By focusing on adaptiveness during its development, VALID encompasses 331 unique checklist versions, making it a one-stop solution suitable for a wide range of projects, designs, and requirements.",

keywords = "Delphi study, Method reform, Open data, Open materials, Open practices, Preregistered, Study quality appraisal, Tailored online checklist, Validity",

author = "Susanne Kerschbaumer and Martin Voracek and Balazs Aczel and Anderson, \{Samantha F.\} and Booth, \{Brandon M.\} and Buchanan, \{Erin M.\} and Rickard Carlsson and Heck, \{Daniel W.\} and Hiekkaranta, \{Anu Pauliina\} and Rink Hoekstra and Karch, \{Julian D.\} and Ginette Lafit and Zhicheng Lin and Siwei Liu and Mackinnon, \{David P.\} and Mcgorray, \{Emma L.\} and David Moreau and Marietta Papadatou-Pastou and Helena Paterson and Perera, \{Robert A.\} and Schad, \{Daniel J.\} and Sewell, \{David K.\} and Moin Syed and Louis Tay and Tendeiro, \{Jorge N.\} and Toland, \{Michael D.\} and Wolf Vanpaemel and \{van Ravenzwaaij\}, Don and Lieke Voncken and Tran, \{Ulrich S.\}",

note = "Additional supporting information can be found at http://journals.sagepub.com/doi/supp10.1177/25152459241306432.",

year = "2025",

month = mar,

doi = "10.1177/25152459241306432",

language = "English",

volume = "8",

journal = "Advances in Methods and Practices in Psychological Science",

issn = "2515-2459",

publisher = "SAGE Publications Inc.",

number = "1",

}

Kerschbaumer, S, Voracek, M, Aczel, B, Anderson, SF, Booth, BM, Buchanan, EM, Carlsson, R, Heck, DW, Hiekkaranta, AP, Hoekstra, R, Karch, JD, Lafit, G, Lin, Z, Liu, S, Mackinnon, DP, Mcgorray, EL, Moreau, D, Papadatou-Pastou, M, Paterson, H, Perera, RA, Schad, DJ, Sewell, DK, Syed, M, Tay, L, Tendeiro, JN, Toland, MD, Vanpaemel, W, van Ravenzwaaij, D, Voncken, L & Tran, US 2025, 'VALID: A checklist-based approach for improving validity in psychological research', Advances in Methods and Practices in Psychological Science, vol. 8, no. 1, 25152459241306432. https://doi.org/10.1177/25152459241306432

TY - JOUR

T1 - VALID

T2 - A checklist-based approach for improving validity in psychological research

AU - Kerschbaumer, Susanne

AU - Voracek, Martin

AU - Aczel, Balazs

AU - Anderson, Samantha F.

AU - Booth, Brandon M.

AU - Buchanan, Erin M.

AU - Carlsson, Rickard

AU - Heck, Daniel W.

AU - Hiekkaranta, Anu Pauliina

AU - Hoekstra, Rink

AU - Karch, Julian D.

AU - Lafit, Ginette

AU - Lin, Zhicheng

AU - Liu, Siwei

AU - Mackinnon, David P.

AU - Mcgorray, Emma L.

AU - Moreau, David

AU - Papadatou-Pastou, Marietta

AU - Paterson, Helena

AU - Perera, Robert A.

AU - Schad, Daniel J.

AU - Sewell, David K.

AU - Syed, Moin

AU - Tay, Louis

AU - Tendeiro, Jorge N.

AU - Toland, Michael D.

AU - Vanpaemel, Wolf

AU - van Ravenzwaaij, Don

AU - Voncken, Lieke

AU - Tran, Ulrich S.

N1 - Additional supporting information can be found at http://journals.sagepub.com/doi/supp10.1177/25152459241306432.

PY - 2025/3

Y1 - 2025/3

N2 - In response to the replication and confidence crisis across various empirical disciplines, ensuring the validity of research has gained attention. High validity is crucial for obtaining replicable and robust study outcomes when both exploring new questions and replicating previous findings. In this study, we aimed to address this issue by developing a comprehensive checklist to assist researchers in enhancing and monitoring the validity of their research. After systematically analyzing previous findings on validity, a comprehensive list of potential checklist items was compiled. Over the course of three rounds, more than 30 interdisciplinary and psychological-science experts participated in a Delphi study. Experts rated items on their importance and were given the opportunity to propose novel items as well as improve existing ones. This process resulted in a final set of 91 items, organized according to common stages of a research project. The VALID checklist is accessible online (https://www.validchecklist.com/) and provides researchers with an adaptable, versatile tool to monitor and improve the validity of their research and to suit their specific needs. By focusing on adaptiveness during its development, VALID encompasses 331 unique checklist versions, making it a one-stop solution suitable for a wide range of projects, designs, and requirements.

AB - In response to the replication and confidence crisis across various empirical disciplines, ensuring the validity of research has gained attention. High validity is crucial for obtaining replicable and robust study outcomes when both exploring new questions and replicating previous findings. In this study, we aimed to address this issue by developing a comprehensive checklist to assist researchers in enhancing and monitoring the validity of their research. After systematically analyzing previous findings on validity, a comprehensive list of potential checklist items was compiled. Over the course of three rounds, more than 30 interdisciplinary and psychological-science experts participated in a Delphi study. Experts rated items on their importance and were given the opportunity to propose novel items as well as improve existing ones. This process resulted in a final set of 91 items, organized according to common stages of a research project. The VALID checklist is accessible online (https://www.validchecklist.com/) and provides researchers with an adaptable, versatile tool to monitor and improve the validity of their research and to suit their specific needs. By focusing on adaptiveness during its development, VALID encompasses 331 unique checklist versions, making it a one-stop solution suitable for a wide range of projects, designs, and requirements.

KW - Delphi study

KW - Method reform

KW - Open data

KW - Open materials

KW - Open practices

KW - Preregistered

KW - Study quality appraisal

KW - Tailored online checklist

KW - Validity

U2 - 10.1177/25152459241306432

DO - 10.1177/25152459241306432

M3 - Article

SN - 2515-2459

VL - 8

JO - Advances in Methods and Practices in Psychological Science

JF - Advances in Methods and Practices in Psychological Science

IS - 1

M1 - 25152459241306432

ER -

VALID: A checklist-based approach for improving validity in psychological research

Abstract

Keywords

Access to Document

Fingerprint

Cite this