THE EFFECT OF NON-NORMALITY ON THE CUTSCORE OPERATING FUNCTION: ESTIMATION CORRECTNESS IN NON-NORMAL MONTE CARLO SIMULATIONS

Abstract

Certification testing attempts to classify individuals into mutually exclusive categories, such as competent and non-competent. There is some potential for error whenever a classification decision is made as a result of a test score. The Grabovsky and Wainer cutscore operating function (GW-CSOF) is a recent addition to classification error estimates. This method allows for the prediction of error rates at all possible cutscore locations, but requires that certain assumptions about the examinee distribution are met. How the estimates made by the GW-CSOF compare to actual error values is currently unknown. Furthermore, the extent to which deviations from GW-CSOF assumptions impact error estimates is also unknown. The aim of this dissertation was to explore the extent to which non-normality of examinee true scores impacted the correctness of the GW-CSOF estimates. Monte Carlo methods were used to generate true score samples with systematically increased non-normality, and GW-CSOF estimates were compared to actual error rates. Findings indicated that GW-CSOF produced good estimates of error rates and optimal cutscore location in truly normal and minimally non-normal simulations. The degree to which GW-CSOF produced incorrect estimates was significantly correlated with the degree of non-normality. Specific guidelines for standard setting are discussed.