Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires

    Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

    Abstract

    Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.
    LanguageEnglish
    Title of host publicationAssessing contexts of learning world-wide
    Subtitle of host publicationAn international perspective
    EditorsS. Kuger, E. Klieme, N. Jude, D. Kaplan
    Place of PublicationNew York
    PublisherSpringer Science
    Pages229-253
    Number of pages24
    ISBN (Print)9783319453576
    DOIs
    StatePublished - 20 Dec 2016

    Fingerprint

    PISA study
    questionnaire
    trend
    process of teaching
    behavioral science
    equivalence
    educational policy
    psychometrics
    learning
    evidence
    student
    social science

    Cite this

    van de Vijver, F., & He, J. (2016). Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. In S. Kuger, E. Klieme, N. Jude, & D. Kaplan (Eds.), Assessing contexts of learning world-wide: An international perspective (pp. 229-253). New York: Springer Science. DOI: 10.1007/987-3-319-45357-6_9
    van de Vijver, Fons ; He, Jia. / Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. Assessing contexts of learning world-wide: An international perspective . editor / S. Kuger ; E. Klieme ; N. Jude ; D. Kaplan. New York : Springer Science, 2016. pp. 229-253
    @inbook{57eec5165d9347d6a9285039418b0c8b,
    title = "Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires",
    abstract = "Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.",
    author = "{van de Vijver}, Fons and Jia He",
    year = "2016",
    month = "12",
    day = "20",
    doi = "10.1007/987-3-319-45357-6_9",
    language = "English",
    isbn = "9783319453576",
    pages = "229--253",
    editor = "S. Kuger and Klieme, {E. } and N. Jude and Kaplan, {D. }",
    booktitle = "Assessing contexts of learning world-wide",
    publisher = "Springer Science",

    }

    van de Vijver, F & He, J 2016, Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. in S Kuger, E Klieme, N Jude & D Kaplan (eds), Assessing contexts of learning world-wide: An international perspective . Springer Science, New York, pp. 229-253. DOI: 10.1007/987-3-319-45357-6_9

    Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. / van de Vijver, Fons; He, Jia.

    Assessing contexts of learning world-wide: An international perspective . ed. / S. Kuger; E. Klieme; N. Jude; D. Kaplan. New York : Springer Science, 2016. p. 229-253.

    Research output: Chapter in Book/Report/Conference proceedingChapterScientificpeer-review

    TY - CHAP

    T1 - Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires

    AU - van de Vijver,Fons

    AU - He,Jia

    PY - 2016/12/20

    Y1 - 2016/12/20

    N2 - Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.

    AB - Assessing students’ personal characteristics, as well as the structures and processes of teaching and learning, is an integral part of the Programme for International Student Assessment (PISA). Providing input for solid evidence-based educational policies, one of the main aims of PISA, creates huge methodological challenges: Various biases in self-reported data across cultures pose a persistent challenge in unpacking the black box of student learning; these biases jeopardize PISA’s scope for evidence-based policy making. This chapter focuses on challenges in the design and analysis of PISA background questionnaires, especially in noncognitive outcome measures. Our conceptual background however is not primarily PISA-related but is comparative work in the social and behavioral sciences more broadly. We first review sources of bias at construct, method, and item levels, as well as levels of equivalence (construct, metric, and scalar invariance), using examples from educational surveys. We then illustrate the strategies used in the PISA project to deal with different types of bias. Specifically, qualitative, non-statistical strategies such as instrument development (adaptation), standardization of assessment procedures, and statistical strategies to mitigate bias are outlined. State-of-the-art psychometric procedures to examine the comparability of these noncognitive outcome data, including partial invariance and approximate invariance, are also discussed. We conclude by suggesting future research topics.

    U2 - 10.1007/987-3-319-45357-6_9

    DO - 10.1007/987-3-319-45357-6_9

    M3 - Chapter

    SN - 9783319453576

    SP - 229

    EP - 253

    BT - Assessing contexts of learning world-wide

    PB - Springer Science

    CY - New York

    ER -

    van de Vijver F, He J. Bias assessment and prevention in non-cognitive outcome measures in PISA questionnaires. In Kuger S, Klieme E, Jude N, Kaplan D, editors, Assessing contexts of learning world-wide: An international perspective . New York: Springer Science. 2016. p. 229-253. Available from, DOI: 10.1007/987-3-319-45357-6_9