Using Expert Knowledge for Test Linking

Linking and equating procedures are used to make the results of different test forms comparable. In the cases where no assumption of random equivalent groups can be made some form of linking design is used. In practice the amount of data available to link the two tests is often very limited due to logistic and security reasons, which affects the precision of linking procedures. This study proposes to enhance the quality of linking procedures based on sparse data by using Bayesian methods which combine the information in the linking data with background information captured in informative prior distributions. We propose two methods for the elicitation of prior knowledge about the difference in difficulty of two tests from subject-matter experts and explain how these results can be used in the specification of priors. To illustrate the proposed methods and evaluate the quality of linking with and without informative priors, an empirical example of linking primary school mathematics tests is presented. The results suggest that informative priors can increase the precision of linking without decreasing the accuracy.

Translational Abstract

If each year a new version of an educational test is used then the results of the new version are not directly comparable to the results of the reference test version due to the difference in difficulty of the 2 tests and the differences in the ability of the new and reference populations of students. Typically extra data ( other than the examination data) are collected to compare the difficulty of the items in the two tests. However, in high stakes testing, where the amount of data that are available to link the reference and the new test are limited due to security reasons, these extra data do not provide enough information to obtain the desired levels of certainty. In this article we argue that linking data are not the only source of information about the difference in the difficulty of the two test forms. Experts may also provide information about this difference. In the study we propose and evaluate two methods for elicitation of the prior knowledge about the difference in difficulty of two tests from subject-matter experts. The results prove the utility of the proposed methodology, since the precision of the linking results increases without the increase in bias.