Adaptive pairwise comparison for educational measurement

Elise Crompvoets, A.A. Béguin, K. Sijtsma

Research output: Contribution to journalArticleScientificpeer-review

2 Downloads (Pure)

Abstract

Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informative comparisons while taking the uncertainty of the object parameters into account. The results of the simulation study showed that, given the number of comparisons, the ASA resulted in smaller standard errors of object parameter estimates than a random selection algorithm that served as a benchmark. Rank order accuracy and reliability were similar for the two algorithms. Because the scale separation reliability (SSR) may overestimate the benchmark reliability when the ASA is used, caution is required when interpreting the SSR.
Original languageEnglish
Number of pages23
JournalJournal of Educational and Behavioral Statistics
DOIs
Publication statusE-pub ahead of print - 2020

Fingerprint

measurement method
uncertainty
simulation
education

Keywords

  • adaptive measurement
  • comparative judgment
  • holistic measurement
  • pairwise comparison

Cite this

@article{4336abe306614cb58cc5b296e13b8980,
title = "Adaptive pairwise comparison for educational measurement",
abstract = "Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informative comparisons while taking the uncertainty of the object parameters into account. The results of the simulation study showed that, given the number of comparisons, the ASA resulted in smaller standard errors of object parameter estimates than a random selection algorithm that served as a benchmark. Rank order accuracy and reliability were similar for the two algorithms. Because the scale separation reliability (SSR) may overestimate the benchmark reliability when the ASA is used, caution is required when interpreting the SSR.",
keywords = "adaptive measurement, comparative judgment, holistic measurement, pairwise comparison",
author = "Elise Crompvoets and A.A. B{\'e}guin and K. Sijtsma",
year = "2020",
doi = "10.3102/1076998619890589",
language = "English",
journal = "Journal of Educational and Behavioral Statistics",
issn = "1076-9986",
publisher = "Sage Publications, Inc.",

}

Adaptive pairwise comparison for educational measurement. / Crompvoets, Elise; Béguin, A.A.; Sijtsma, K.

In: Journal of Educational and Behavioral Statistics, 2020.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Adaptive pairwise comparison for educational measurement

AU - Crompvoets, Elise

AU - Béguin, A.A.

AU - Sijtsma, K.

PY - 2020

Y1 - 2020

N2 - Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informative comparisons while taking the uncertainty of the object parameters into account. The results of the simulation study showed that, given the number of comparisons, the ASA resulted in smaller standard errors of object parameter estimates than a random selection algorithm that served as a benchmark. Rank order accuracy and reliability were similar for the two algorithms. Because the scale separation reliability (SSR) may overestimate the benchmark reliability when the ASA is used, caution is required when interpreting the SSR.

AB - Pairwise comparison is becoming increasingly popular as a holistic measurement method in education. Unfortunately, many comparisons are required for reliable measurement. To reduce the number of required comparisons, we developed an adaptive selection algorithm (ASA) that selects the most informative comparisons while taking the uncertainty of the object parameters into account. The results of the simulation study showed that, given the number of comparisons, the ASA resulted in smaller standard errors of object parameter estimates than a random selection algorithm that served as a benchmark. Rank order accuracy and reliability were similar for the two algorithms. Because the scale separation reliability (SSR) may overestimate the benchmark reliability when the ASA is used, caution is required when interpreting the SSR.

KW - adaptive measurement

KW - comparative judgment

KW - holistic measurement

KW - pairwise comparison

U2 - 10.3102/1076998619890589

DO - 10.3102/1076998619890589

M3 - Article

JO - Journal of Educational and Behavioral Statistics

JF - Journal of Educational and Behavioral Statistics

SN - 1076-9986

ER -