Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires

Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

Abstract

Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.
LanguageEnglish
Title of host publicationAssessing contexts of learning world-wide
Subtitle of host publicationAn international perspective
EditorsS. Kuger, E. Klieme, N. Jude, D. Kaplan
Place of PublicationNew York
PublisherSpringer Science
Pages229-253
Number of pages24
ISBN (Print)9783319453576
DOIs
Publication statusPublished - 20 Dec 2016

Fingerprint

PISA study
questionnaire
trend
process of teaching
behavioral science
equivalence
educational policy
psychometrics
learning
evidence
student
social science

Cite this

van de Vijver, F., & He, J. (2016). Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning world-wide: An international perspective (pp. 229-253). New York: Springer Science. https://doi.org/10.1007/987-3-319-45357-6_9
van de Vijver, Fons ; He, Jia. / Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. Assessing contexts of learning world-wide: An international perspective . editor / S. Kuger ; E. Klieme ; N. Jude ; D. Kaplan. New York : Springer Science, 2016. pp. 229-253
@inbook{57eec5165d9347d6a9285039418b0c8b,
title = "Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires",
abstract = "Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.",
author = "{van de Vijver}, Fons and Jia He",
year = "2016",
month = "12",
day = "20",
doi = "10.1007/987-3-319-45357-6_9",
language = "English",
isbn = "9783319453576",
pages = "229--253",
editor = "S. Kuger and Klieme, {E. } and N. Jude and Kaplan, {D. }",
booktitle = "Assessing contexts of learning world-wide",
publisher = "Springer Science",

}

van de Vijver, F & He, J 2016, Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. in S Kuger, E Klieme, N Jude & D Kaplan (eds), Assessing contexts of learning world-wide: An international perspective . Springer Science, New York, pp. 229-253. https://doi.org/10.1007/987-3-319-45357-6_9

Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. / van de Vijver, Fons; He, Jia.

Assessing contexts of learning world-wide: An international perspective . ed. / S. Kuger; E. Klieme; N. Jude; D. Kaplan. New York : Springer Science, 2016. p. 229-253.

Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

TY - CHAP

T1 - Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires

AU - van de Vijver, Fons

AU - He, Jia

PY - 2016/12/20

Y1 - 2016/12/20

N2 - Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.

AB - Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.

U2 - 10.1007/987-3-319-45357-6_9

DO - 10.1007/987-3-319-45357-6_9

M3 - Chapter

SN - 9783319453576

SP - 229

EP - 253

BT - Assessing contexts of learning world-wide

A2 - Kuger, S.

A2 - Klieme, E.

A2 - Jude, N.

A2 - Kaplan, D.

PB - Springer Science

CY - New York

ER -

van de Vijver F, He J. Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. In Kuger S, Klieme E, Jude N, Kaplan D, editors, Assessing contexts of learning world-wide: An international perspective . New York: Springer Science. 2016. p. 229-253 https://doi.org/10.1007/987-3-319-45357-6_9