Tracking with (un)certainty

Abe D. Hofman*, Matthieu J. S. Brinkhuis, Maria Bolsinova, Jonathan Klaiber, Gunter Maris, Han L. J. van der Maas

*Corresponding author for this work

Research output: Contribution to journalArticleScientificpeer-review

24 Downloads (Pure)


One of the highest ambitions in educational technology is the move towards personalized learning. To this end, computerized adaptive learning (CAL) systems are developed. A popular method to track the development of student ability and item difficulty, in CAL systems, is the Elo Rating System (ERS). The ERS allows for dynamic model parameters by updating key parameters after every response. However, drawbacks of the ERS are that it does not provide standard errors and that it results in rating variance inflation. We identify three statistical issues responsible for both of these drawbacks. To solve these issues we introduce a new tracking system based on urns, where every person and item is represented by an urn filled with a combination of green and red marbles. Urns are updated, by an exchange of marbles after each response, such that the proportions of green marbles represent estimates of person ability or item difficulty. A main advantage of this approach is that the standard errors are known, hence the method allows for statistical inference, such as testing for learning effects. We highlight features of the Urnings algorithm and compare it to the popular ERS in a simulation study and in an empirical data example from a large-scale CAL application.

Original languageEnglish
Article number10
Number of pages15
JournalJournal of Intelligence
Issue number1
Publication statusPublished - 2020


  • computerized adaptive learning systems
  • student modelling
  • tracking
  • statistical inferences


Dive into the research topics of 'Tracking with (un)certainty'. Together they form a unique fingerprint.

Cite this