Tracing systematic errors to personalize recommendations in single digit multiplication and beyond

In learning, errors are ubiquitous and inevitable. As these errors may signal otherwise latent cognitive processes, tutors - and students alike - can greatly benefit from the information they provide. In this paper, we introduce and evaluate the Systematic Error Tracing (SET) model that identifies the possible causes of systematically observed errors in domains where items are susceptible to most or all causes and errors can be explained by multiple causes. We apply the model to single-digit multiplication, a domain that is very suitable for the model, is well-studied, and allows us to analyze over 25,000 error responses from 335 learners. The model, derived from the Ising model popular in physics, makes use of a bigraph that links errors to causes. The error responses were taken from Math Garden, a computerized adaptive practice environment for arithmetic that is widely used in the Netherlands. We discuss and evaluate various model configurations with respect to the ranking of recommendations and calibration of probability estimates. The results show that the SET model outranks a majority vote baseline model when more than a single recommendation is considered. Finally, we contrast the SET model to similar approaches and discuss limitations and implications.