Improving precision of ability estimation

Getting more from response times

Maria Bolsinova, J. Tijmstra

Research output: Contribution to journalArticleScientificpeer-review

Abstract

By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross‐loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA‐only model is considered. The model is applied to data from a high‐stakes educational test.
Original languageEnglish
Pages (from-to)13-38
JournalBritish Journal of Mathematical and Statistical Psychology
Volume71
Issue number1
DOIs
Publication statusPublished - 2018

Fingerprint

Hierarchical Model
Response Time
Conditional Independence
Standard Model
Joint Model
Model
Bayesian Estimation
Person
Recovery
Simulation Study

Keywords

  • ACCURACY
  • CONDITIONAL-INDEPENDENCE
  • DISTRIBUTIONS
  • FIT
  • FRAMEWORK
  • TESTS
  • THEORY MODELS
  • collateral information
  • hierarchical model
  • item response theory
  • measurement precision
  • response times

Cite this

@article{0fae33a98c6d46a89e8f3be6b8d61d5d,
title = "Improving precision of ability estimation: Getting more from response times",
abstract = "By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross‐loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA‐only model is considered. The model is applied to data from a high‐stakes educational test.",
keywords = "ACCURACY, CONDITIONAL-INDEPENDENCE, DISTRIBUTIONS, FIT, FRAMEWORK, TESTS, THEORY MODELS, collateral information, hierarchical model, item response theory, measurement precision, response times",
author = "Maria Bolsinova and J. Tijmstra",
year = "2018",
doi = "10.1111/bmsp.12104",
language = "English",
volume = "71",
pages = "13--38",
journal = "British Journal of Mathematical and Statistical Psychology",
issn = "0007-1102",
publisher = "Wiley",
number = "1",

}

Improving precision of ability estimation : Getting more from response times. / Bolsinova, Maria; Tijmstra, J.

In: British Journal of Mathematical and Statistical Psychology, Vol. 71, No. 1, 2018, p. 13-38.

Research output: Contribution to journalArticleScientificpeer-review

TY - JOUR

T1 - Improving precision of ability estimation

T2 - Getting more from response times

AU - Bolsinova, Maria

AU - Tijmstra, J.

PY - 2018

Y1 - 2018

N2 - By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross‐loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA‐only model is considered. The model is applied to data from a high‐stakes educational test.

AB - By considering information about response time (RT) in addition to response accuracy (RA), joint models for RA and RT such as the hierarchical model (van der Linden, 2007) can improve the precision with which ability is estimated over models that only consider RA. The hierarchical model, however, assumes that only the person's speed is informative of ability. This assumption of conditional independence between RT and ability given speed may be violated in practice, and ignores collateral information about ability that may be present in the residual RTs. We propose a posterior predictive check for evaluating the assumption of conditional independence between RT and ability given speed. Furthermore, we propose an extension of the hierarchical model that contains cross‐loadings between ability and RT, which enables one to take additional collateral information about ability into account beyond what is possible in the standard hierarchical model. A Bayesian estimation procedure is proposed for the model. Using simulation studies, the performance of the model is evaluated in terms of parameter recovery, and the possible gain in precision over the standard hierarchical model and an RA‐only model is considered. The model is applied to data from a high‐stakes educational test.

KW - ACCURACY

KW - CONDITIONAL-INDEPENDENCE

KW - DISTRIBUTIONS

KW - FIT

KW - FRAMEWORK

KW - TESTS

KW - THEORY MODELS

KW - collateral information

KW - hierarchical model

KW - item response theory

KW - measurement precision

KW - response times

U2 - 10.1111/bmsp.12104

DO - 10.1111/bmsp.12104

M3 - Article

VL - 71

SP - 13

EP - 38

JO - British Journal of Mathematical and Statistical Psychology

JF - British Journal of Mathematical and Statistical Psychology

SN - 0007-1102

IS - 1

ER -