Item-score reliability as a selection tool in test construction

E.A.O. Zijlmans*, J. Tijmstra, L.A. van der Ark, K. Sijtsma

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

4 Downloads (Pure)

Abstract

This study investigates the usefulness of item-score reliability as a criterion for item selection in test construction. Methods MS, λ 6, and CA were investigated as item-assessment methods in item selection and compared to the corrected item-total correlation, which was used as a benchmark. An ideal ordering to add items to the test (bottom-up procedure) or omit items from the test (top-down procedure) was defined based on the population test-score reliability. The orderings the four item-assessment methods produced in samples were compared to the ideal ordering, and the degree of resemblance was expressed by means of Kendall's τ. To investigate the concordance of the orderings across 1,000 replicated samples, Kendall's W was computed for each item-assessment method. The results showed that for both the bottom-up and the top-down procedures, item-assessment method CA and the corrected item-total correlation most closely resembled the ideal ordering. Generally, all item assessment methods resembled the ideal ordering better, and concordance of the orderings was greater, for larger sample sizes, and greater variance of the item discrimination parameters.

Original languageEnglish
Article number2298
Number of pages12
JournalFrontiers in Psychology
Volume2018
Issue number9
DOIs
Publication statusPublished - 2019

Fingerprint

Benchmarking
Patient Selection

Keywords

  • ALPHA
  • SATISFACTION
  • SCALE
  • SINGLE-ITEM
  • VALIDITY
  • corrected item-total correlation
  • correction for attenuation
  • item selection in test construction
  • item-score reliability
  • method CA
  • method MS
  • method lambda(6)

Cite this

Zijlmans, E.A.O. ; Tijmstra, J. ; van der Ark, L.A. ; Sijtsma, K. / Item-score reliability as a selection tool in test construction. In: Frontiers in Psychology. 2019 ; Vol. 2018, No. 9.
@article{22b4f9132c6b4b57a8f7751f8635015e,
title = "Item-score reliability as a selection tool in test construction",
abstract = "This study investigates the usefulness of item-score reliability as a criterion for item selection in test construction. Methods MS, λ 6, and CA were investigated as item-assessment methods in item selection and compared to the corrected item-total correlation, which was used as a benchmark. An ideal ordering to add items to the test (bottom-up procedure) or omit items from the test (top-down procedure) was defined based on the population test-score reliability. The orderings the four item-assessment methods produced in samples were compared to the ideal ordering, and the degree of resemblance was expressed by means of Kendall's τ. To investigate the concordance of the orderings across 1,000 replicated samples, Kendall's W was computed for each item-assessment method. The results showed that for both the bottom-up and the top-down procedures, item-assessment method CA and the corrected item-total correlation most closely resembled the ideal ordering. Generally, all item assessment methods resembled the ideal ordering better, and concordance of the orderings was greater, for larger sample sizes, and greater variance of the item discrimination parameters.",
keywords = "ALPHA, SATISFACTION, SCALE, SINGLE-ITEM, VALIDITY, corrected item-total correlation, correction for attenuation, item selection in test construction, item-score reliability, method CA, method MS, method lambda(6)",
author = "E.A.O. Zijlmans and J. Tijmstra and {van der Ark}, L.A. and K. Sijtsma",
year = "2019",
doi = "10.3389/fpsyg.2018.02298",
language = "English",
volume = "2018",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Media S.A.",
number = "9",

}

Item-score reliability as a selection tool in test construction. / Zijlmans, E.A.O.; Tijmstra, J.; van der Ark, L.A.; Sijtsma, K.

In: Frontiers in Psychology, Vol. 2018, No. 9, 2298, 2019.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Item-score reliability as a selection tool in test construction

AU - Zijlmans, E.A.O.

AU - Tijmstra, J.

AU - van der Ark, L.A.

AU - Sijtsma, K.

PY - 2019

Y1 - 2019

N2 - This study investigates the usefulness of item-score reliability as a criterion for item selection in test construction. Methods MS, λ 6, and CA were investigated as item-assessment methods in item selection and compared to the corrected item-total correlation, which was used as a benchmark. An ideal ordering to add items to the test (bottom-up procedure) or omit items from the test (top-down procedure) was defined based on the population test-score reliability. The orderings the four item-assessment methods produced in samples were compared to the ideal ordering, and the degree of resemblance was expressed by means of Kendall's τ. To investigate the concordance of the orderings across 1,000 replicated samples, Kendall's W was computed for each item-assessment method. The results showed that for both the bottom-up and the top-down procedures, item-assessment method CA and the corrected item-total correlation most closely resembled the ideal ordering. Generally, all item assessment methods resembled the ideal ordering better, and concordance of the orderings was greater, for larger sample sizes, and greater variance of the item discrimination parameters.

AB - This study investigates the usefulness of item-score reliability as a criterion for item selection in test construction. Methods MS, λ 6, and CA were investigated as item-assessment methods in item selection and compared to the corrected item-total correlation, which was used as a benchmark. An ideal ordering to add items to the test (bottom-up procedure) or omit items from the test (top-down procedure) was defined based on the population test-score reliability. The orderings the four item-assessment methods produced in samples were compared to the ideal ordering, and the degree of resemblance was expressed by means of Kendall's τ. To investigate the concordance of the orderings across 1,000 replicated samples, Kendall's W was computed for each item-assessment method. The results showed that for both the bottom-up and the top-down procedures, item-assessment method CA and the corrected item-total correlation most closely resembled the ideal ordering. Generally, all item assessment methods resembled the ideal ordering better, and concordance of the orderings was greater, for larger sample sizes, and greater variance of the item discrimination parameters.

KW - ALPHA

KW - SATISFACTION

KW - SCALE

KW - SINGLE-ITEM

KW - VALIDITY

KW - corrected item-total correlation

KW - correction for attenuation

KW - item selection in test construction

KW - item-score reliability

KW - method CA

KW - method MS

KW - method lambda(6)

U2 - 10.3389/fpsyg.2018.02298

DO - 10.3389/fpsyg.2018.02298

M3 - Article

VL - 2018

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

IS - 9

M1 - 2298

ER -