THE IMPACT OF ANCHOR ITEM EXPOSURE ON MEAN/SIGMA LINKING AND IRT TRUE SCORE EQUATING UNDER THE NEAT DESIGN

Abstract

To compare examinees' true ability and their actual competence on the content being measured across different test administrations, test scores must be equated. One of the most common equating designs is called the nonequivalent anchor test (NEAT) design. This equating design requires two forms of a test, each of which is given to a group of examinees one year apart. The two forms have a set of items in common, usually called the anchor set, in order to control for differences in examinee ability. The anchor set can be treated as internal or external according to whether or not examinees' responses contribute to their total score. However, the anchor set is subject to exposure when it is used repeatedly, which most likely becomes a serious threat to test fairness and validity. Therefore, from time to time, the items in the anchor set must be evaluated for exposure. This study employed a Monte Carlo investigation to evaluate the impact of internal anchor item exposure on the equating process under the NEAT design. The study addressed a general scenario in which two forms of a small-scale dichotomously scored test were given to two small groups of examinees. Since mean/sigma linking and true score equating are main components of the equating process in the item response theory (IRT), the recovery of equating true scores and linking coefficients, slope and intercept, were assessed under various combinations of testing conditions using bias and mean squared error (MSE). Three testing conditions were manipulated in this study: (a) the number of exposed anchor items, (b) the percentage of examinees with preknowledge of the exposed anchor items, and (c) the difference in the means of ability distributions of groups taking the original form and new form. In each combination of testing conditions, the simulation process was replicated 100 times. The study results indicated that anchor item exposure caused all examinees to receive inflated equating true scores. When anchor items were subject to low levels of exposure, the accuracy of equating true scores was still perturbing, while high levels of exposure distorted the test scores completely. The anchor item exposure became a serious threat to the test fairness to the extent that unqualified examinees might receive an unfair benefit over qualified examinees who completed an unexposed test form.