On detecting systematic measurement error in cross-cultural research: A review and critical reflection on equivalence and invariance tests

Diana Boer, Katja Hanke, J. He

Research output: Contribution to journalArticleScientificpeer-review

Abstract

One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.
LanguageEnglish
Pages713-734
JournalJournal of Cross-Cultural Psychology
Volume49
Issue number5
DOIs
StatePublished - 1 Jun 2018

Fingerprint

equivalence
trend
content analysis
cultural psychology
developmental psychology
social psychology
threat
paradigm

Cite this

@article{a6b3899bad8a4bb5bc4f643640fd0216,
title = "On detecting systematic measurement error in cross-cultural research: A review and critical reflection on equivalence and invariance tests",
abstract = "One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.",
author = "Diana Boer and Katja Hanke and J. He",
year = "2018",
month = "6",
day = "1",
doi = "10.1177/0022022117749042",
language = "English",
volume = "49",
pages = "713--734",
journal = "Journal of Cross-Cultural Psychology",
issn = "0022-0221",
publisher = "Sage Publications, Inc.",
number = "5",

}

On detecting systematic measurement error in cross-cultural research : A review and critical reflection on equivalence and invariance tests. / Boer, Diana; Hanke, Katja; He, J.

In: Journal of Cross-Cultural Psychology, Vol. 49, No. 5, 01.06.2018, p. 713-734.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - On detecting systematic measurement error in cross-cultural research

T2 - Journal of Cross-Cultural Psychology

AU - Boer,Diana

AU - Hanke,Katja

AU - He,J.

PY - 2018/6/1

Y1 - 2018/6/1

N2 - One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.

AB - One major threat to revealing cultural influences on psychological states or processes is the presence of bias (i.e., systematic measurement error). When quantitative measures are not targeting the same construct or they differ in metric across cultures, the validity of inferences about cultural variability (and universality) is in doubt. The objectives of this article are to review what can be done about it and what is being done about it. To date, a multitude of useful techniques and methods to reduce or assess bias in cross-cultural research have been developed. We explore the limits of invariance/equivalence testing and suggest more flexible means of dealing with bias. First, we review currently available established and novel methods that reveal bias in cross-cultural research. Second, we analyze current practices in a systematic content analysis. The content analysis of more than 500 culture-comparative quantitative studies (published from 2008 to 2015 in three outlets in cross-cultural, social, and developmental psychology) aims to gauge current practices and approaches in the assessment of measurement equivalence/invariance. Surprisingly, the analysis revealed a rather low penetration of invariance testing in cross-cultural research. Although a multitude of classical and novel approaches for invariance testing is available, these are employed infrequent rather than habitual. We discuss reasons for this hesitation, and we derive suggestions for creatively assessing and handling biases across different research paradigms and designs.

U2 - 10.1177/0022022117749042

DO - 10.1177/0022022117749042

M3 - Article

VL - 49

SP - 713

EP - 734

JO - Journal of Cross-Cultural Psychology

JF - Journal of Cross-Cultural Psychology

SN - 0022-0221

IS - 5

ER -