Assessing the equivalence of web-based and paper-and-pencil questionnaires using differential item and test functioning (DIF and DTF) analysis

A case of the Four-Dimensional Symptom Questionnaire (4DSQ)

B. Terluin, E.P.M. Brouwers, M.A.G. Marchand, H.C. De Vet

Research output: Contribution to journalArticleScientificpeer-review

28 Downloads (Pure)

Abstract

Purpose: 
Many paper-and-pencil (P&P) questionnaires have been migrated to electronic platforms. Differential item and test functioning (DIF and DTF) analysis constitutes a superior research design to assess measurement equivalence across modes of administration. The purpose of this study was to demonstrate an item response theory (IRT)-based DIF and DTF analysis to assess the measurement equivalence of a Web-based version and the original P&P format of the Four-Dimensional Symptom Questionnaire (4DSQ), measuring distress, depression, anxiety, and somatization.
Methods: 
The P&P group (n=2031) and the Web group (n=958) consisted of primary care psychology clients. Unidimensionality and local independence of the 4DSQ scales were examined using IRT and Yen’s Q3. Bifactor modeling was used to assess the scales’ essential unidimensionality. Measurement equivalence was assessed using IRT-based DIF analysis using a 3-stage approach: linking on the latent mean and variance, selection of anchor items, and DIF testing using the Wald test. DTF was evaluated by comparing expected scale scores as a function of the latent trait.
Results: 
The 4DSQ scales proved to be essentially unidimensional in both modalities. Five items, belonging to the distress and somatization scales, displayed small amounts of DIF. DTF analysis revealed that the impact of DIF on the scale levelwas negligible.
Conclusions: 
IRT-based DIF and DTF analysis is demonstrated as a way to assess the equivalence of Web-based and P&P questionnaire modalities. Data obtained with the Web-based 4DSQ are equivalent to data obtained with the P&P version.
Original languageEnglish
Pages (from-to)1191-1200
JournalQuality of Life Research
Volume27
Issue number5
DOIs
Publication statusPublished - 2018

Fingerprint

Depression
Surveys and Questionnaires

Keywords

  • Bifactor model
  • DEPRESSION
  • Differential item functioning
  • Differential test functioning
  • FIT
  • INDEXES
  • IRT
  • MEASUREMENT INVARIANCE
  • MODELING APPROACH
  • MULTIDIMENSIONALITY
  • Measurement equivalence
  • POPULATION
  • Paper-and-pencil questionnaire
  • REPORTED OUTCOME MEASURES
  • RESPONSE THEORY
  • Web-based questionnaire

Cite this

@article{513827d85ff84c0b841c04afea021ace,
title = "Assessing the equivalence of web-based and paper-and-pencil questionnaires using differential item and test functioning (DIF and DTF) analysis: A case of the Four-Dimensional Symptom Questionnaire (4DSQ)",
abstract = "Purpose: Many paper-and-pencil (P&P) questionnaires have been migrated to electronic platforms. Differential item and test functioning (DIF and DTF) analysis constitutes a superior research design to assess measurement equivalence across modes of administration. The purpose of this study was to demonstrate an item response theory (IRT)-based DIF and DTF analysis to assess the measurement equivalence of a Web-based version and the original P&P format of the Four-Dimensional Symptom Questionnaire (4DSQ), measuring distress, depression, anxiety, and somatization.Methods: The P&P group (n=2031) and the Web group (n=958) consisted of primary care psychology clients. Unidimensionality and local independence of the 4DSQ scales were examined using IRT and Yen’s Q3. Bifactor modeling was used to assess the scales’ essential unidimensionality. Measurement equivalence was assessed using IRT-based DIF analysis using a 3-stage approach: linking on the latent mean and variance, selection of anchor items, and DIF testing using the Wald test. DTF was evaluated by comparing expected scale scores as a function of the latent trait.Results: The 4DSQ scales proved to be essentially unidimensional in both modalities. Five items, belonging to the distress and somatization scales, displayed small amounts of DIF. DTF analysis revealed that the impact of DIF on the scale levelwas negligible.Conclusions: IRT-based DIF and DTF analysis is demonstrated as a way to assess the equivalence of Web-based and P&P questionnaire modalities. Data obtained with the Web-based 4DSQ are equivalent to data obtained with the P&P version.",
keywords = "Bifactor model, DEPRESSION, Differential item functioning, Differential test functioning, FIT, INDEXES, IRT, MEASUREMENT INVARIANCE, MODELING APPROACH, MULTIDIMENSIONALITY, Measurement equivalence, POPULATION, Paper-and-pencil questionnaire, REPORTED OUTCOME MEASURES, RESPONSE THEORY, Web-based questionnaire",
author = "B. Terluin and E.P.M. Brouwers and M.A.G. Marchand and {De Vet}, H.C.",
year = "2018",
doi = "10.1007/s11136-018-1816-5",
language = "English",
volume = "27",
pages = "1191--1200",
journal = "Quality of Life Research",
issn = "0962-9343",
publisher = "Springer",
number = "5",

}

Assessing the equivalence of web-based and paper-and-pencil questionnaires using differential item and test functioning (DIF and DTF) analysis : A case of the Four-Dimensional Symptom Questionnaire (4DSQ). / Terluin, B.; Brouwers, E.P.M.; Marchand, M.A.G.; De Vet, H.C.

In: Quality of Life Research, Vol. 27, No. 5, 2018, p. 1191-1200.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Assessing the equivalence of web-based and paper-and-pencil questionnaires using differential item and test functioning (DIF and DTF) analysis

T2 - A case of the Four-Dimensional Symptom Questionnaire (4DSQ)

AU - Terluin, B.

AU - Brouwers, E.P.M.

AU - Marchand, M.A.G.

AU - De Vet, H.C.

PY - 2018

Y1 - 2018

N2 - Purpose: Many paper-and-pencil (P&P) questionnaires have been migrated to electronic platforms. Differential item and test functioning (DIF and DTF) analysis constitutes a superior research design to assess measurement equivalence across modes of administration. The purpose of this study was to demonstrate an item response theory (IRT)-based DIF and DTF analysis to assess the measurement equivalence of a Web-based version and the original P&P format of the Four-Dimensional Symptom Questionnaire (4DSQ), measuring distress, depression, anxiety, and somatization.Methods: The P&P group (n=2031) and the Web group (n=958) consisted of primary care psychology clients. Unidimensionality and local independence of the 4DSQ scales were examined using IRT and Yen’s Q3. Bifactor modeling was used to assess the scales’ essential unidimensionality. Measurement equivalence was assessed using IRT-based DIF analysis using a 3-stage approach: linking on the latent mean and variance, selection of anchor items, and DIF testing using the Wald test. DTF was evaluated by comparing expected scale scores as a function of the latent trait.Results: The 4DSQ scales proved to be essentially unidimensional in both modalities. Five items, belonging to the distress and somatization scales, displayed small amounts of DIF. DTF analysis revealed that the impact of DIF on the scale levelwas negligible.Conclusions: IRT-based DIF and DTF analysis is demonstrated as a way to assess the equivalence of Web-based and P&P questionnaire modalities. Data obtained with the Web-based 4DSQ are equivalent to data obtained with the P&P version.

AB - Purpose: Many paper-and-pencil (P&P) questionnaires have been migrated to electronic platforms. Differential item and test functioning (DIF and DTF) analysis constitutes a superior research design to assess measurement equivalence across modes of administration. The purpose of this study was to demonstrate an item response theory (IRT)-based DIF and DTF analysis to assess the measurement equivalence of a Web-based version and the original P&P format of the Four-Dimensional Symptom Questionnaire (4DSQ), measuring distress, depression, anxiety, and somatization.Methods: The P&P group (n=2031) and the Web group (n=958) consisted of primary care psychology clients. Unidimensionality and local independence of the 4DSQ scales were examined using IRT and Yen’s Q3. Bifactor modeling was used to assess the scales’ essential unidimensionality. Measurement equivalence was assessed using IRT-based DIF analysis using a 3-stage approach: linking on the latent mean and variance, selection of anchor items, and DIF testing using the Wald test. DTF was evaluated by comparing expected scale scores as a function of the latent trait.Results: The 4DSQ scales proved to be essentially unidimensional in both modalities. Five items, belonging to the distress and somatization scales, displayed small amounts of DIF. DTF analysis revealed that the impact of DIF on the scale levelwas negligible.Conclusions: IRT-based DIF and DTF analysis is demonstrated as a way to assess the equivalence of Web-based and P&P questionnaire modalities. Data obtained with the Web-based 4DSQ are equivalent to data obtained with the P&P version.

KW - Bifactor model

KW - DEPRESSION

KW - Differential item functioning

KW - Differential test functioning

KW - FIT

KW - INDEXES

KW - IRT

KW - MEASUREMENT INVARIANCE

KW - MODELING APPROACH

KW - MULTIDIMENSIONALITY

KW - Measurement equivalence

KW - POPULATION

KW - Paper-and-pencil questionnaire

KW - REPORTED OUTCOME MEASURES

KW - RESPONSE THEORY

KW - Web-based questionnaire

U2 - 10.1007/s11136-018-1816-5

DO - 10.1007/s11136-018-1816-5

M3 - Article

VL - 27

SP - 1191

EP - 1200

JO - Quality of Life Research

JF - Quality of Life Research

SN - 0962-9343

IS - 5

ER -