Modeling Panelist Consistency in Bookmark Standard-Setting Studies

Abstract

The purpose of this study is to develop a psychometrics of standard-setting for the Bookmark standard-setting procedure. Using simulated and real data, the error variance of Bookmark cutscores is modeled as a function of both within-group and between-group variance. Fully Bayesian methods are then used to estimate the total error variance associated with Bookmark cutscores. The results of the study suggest that the estimates of error variance for the proposed method tend to be larger than those calculated using traditional methods. Consequently, the confidence intervals were more likely to include the “true” cutscore in the simulation study. Because the result of a standard-setting study is almost always a recommendation rather than a final cutscore, one goal is to provide policymakers with a more accurate representation of the amount of uncertainty around a panel’s recommendation.