Measurement invariance versus selection invariance: Is fair selection possible?

D. Borsboom, J.W. Romeijn, J.M. Wicherts

Research output: Contribution to journalArticleScientificpeer-review

Abstract

This article shows that measurement invariance (defined in terms of an invariant measurement model in different groups) is generally inconsistent with selection invariance (defined in terms of equal sensitivity and specificity across groups). In particular, when a unidimensional measurement instrument is used and group differences are present in the location but not in the variance of the latent distribution, sensitivity and positive predictive value will be higher in the group at the higher end of the latent dimension, whereas specificity and negative predictive value will be higher in the group at the lower end of the latent dimension. When latent variances are unequal, the differences in these quantities depend on the size of group differences in variances relative to the size of group differences in means. The effect originates as a special case of Simpson’s paradox, which arises because the observed score distribution is collapsed into an accept-reject dichotomy. Simulations show the effect can be substantial in realistic situations. It is suggested that the effect may be partly responsible for overprediction in minority groups as typically found in empirical studies on differential academic performance. A methodological solution to the problem is suggested, and social policy implications are discussed.
Original languageEnglish
Pages (from-to)75-98
Number of pages24
JournalPsychological Methods
Volume13
Issue number2
DOIs
Publication statusPublished - 2008
Externally publishedYes

Fingerprint

Minority Groups

Cite this

@article{79ccb1a7d3694777ae86a20ae99cefaf,
title = "Measurement invariance versus selection invariance: Is fair selection possible?",
abstract = "This article shows that measurement invariance (defined in terms of an invariant measurement model in different groups) is generally inconsistent with selection invariance (defined in terms of equal sensitivity and specificity across groups). In particular, when a unidimensional measurement instrument is used and group differences are present in the location but not in the variance of the latent distribution, sensitivity and positive predictive value will be higher in the group at the higher end of the latent dimension, whereas specificity and negative predictive value will be higher in the group at the lower end of the latent dimension. When latent variances are unequal, the differences in these quantities depend on the size of group differences in variances relative to the size of group differences in means. The effect originates as a special case of Simpson’s paradox, which arises because the observed score distribution is collapsed into an accept-reject dichotomy. Simulations show the effect can be substantial in realistic situations. It is suggested that the effect may be partly responsible for overprediction in minority groups as typically found in empirical studies on differential academic performance. A methodological solution to the problem is suggested, and social policy implications are discussed.",
author = "D. Borsboom and J.W. Romeijn and J.M. Wicherts",
year = "2008",
doi = "10.1037/1082-989X.13.2.75",
language = "English",
volume = "13",
pages = "75--98",
journal = "Psychological Methods",
issn = "1082-989X",
publisher = "AMER PSYCHOLOGICAL ASSOC",
number = "2",

}

Measurement invariance versus selection invariance: Is fair selection possible? / Borsboom, D.; Romeijn, J.W.; Wicherts, J.M.

In: Psychological Methods, Vol. 13, No. 2, 2008, p. 75-98.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Measurement invariance versus selection invariance: Is fair selection possible?

AU - Borsboom, D.

AU - Romeijn, J.W.

AU - Wicherts, J.M.

PY - 2008

Y1 - 2008

N2 - This article shows that measurement invariance (defined in terms of an invariant measurement model in different groups) is generally inconsistent with selection invariance (defined in terms of equal sensitivity and specificity across groups). In particular, when a unidimensional measurement instrument is used and group differences are present in the location but not in the variance of the latent distribution, sensitivity and positive predictive value will be higher in the group at the higher end of the latent dimension, whereas specificity and negative predictive value will be higher in the group at the lower end of the latent dimension. When latent variances are unequal, the differences in these quantities depend on the size of group differences in variances relative to the size of group differences in means. The effect originates as a special case of Simpson’s paradox, which arises because the observed score distribution is collapsed into an accept-reject dichotomy. Simulations show the effect can be substantial in realistic situations. It is suggested that the effect may be partly responsible for overprediction in minority groups as typically found in empirical studies on differential academic performance. A methodological solution to the problem is suggested, and social policy implications are discussed.

AB - This article shows that measurement invariance (defined in terms of an invariant measurement model in different groups) is generally inconsistent with selection invariance (defined in terms of equal sensitivity and specificity across groups). In particular, when a unidimensional measurement instrument is used and group differences are present in the location but not in the variance of the latent distribution, sensitivity and positive predictive value will be higher in the group at the higher end of the latent dimension, whereas specificity and negative predictive value will be higher in the group at the lower end of the latent dimension. When latent variances are unequal, the differences in these quantities depend on the size of group differences in variances relative to the size of group differences in means. The effect originates as a special case of Simpson’s paradox, which arises because the observed score distribution is collapsed into an accept-reject dichotomy. Simulations show the effect can be substantial in realistic situations. It is suggested that the effect may be partly responsible for overprediction in minority groups as typically found in empirical studies on differential academic performance. A methodological solution to the problem is suggested, and social policy implications are discussed.

U2 - 10.1037/1082-989X.13.2.75

DO - 10.1037/1082-989X.13.2.75

M3 - Article

VL - 13

SP - 75

EP - 98

JO - Psychological Methods

JF - Psychological Methods

SN - 1082-989X

IS - 2

ER -