Local Optima in Diagnostic Classification Models

Abstract

A Monte Carol simulation study was used to investigate the prevalence of local optima in Diagnostic classification models (DCMs) under multiple conditions. Five variables were manipulated, including model constraints, starting values, item effect size, attribute correlation, and mastery base rate. Model constraints had two categories (i.e., with and without). Starting values had five categories (i.e., true, random 1, random 2, extremely low, and extremely high). The other three independent variables were continuous, sampling from uniform distributions. Other related variables were fixed at simplified yet reasonable values to avoid interference. There were 1000 replications, each with three attributes, 18 items and 5000 examinees. The simulation design had three levels. Data were at the highest, followed by model constraints, while starting values at the lowest level. For each replication, the same data set was used to estimate parameters 10 times. Half of them were estimated with model constraints. Each had one of those five sets of starting values. The other half were estimated without model constraints, using the same starting values as those in the model with constraints condition. The convergence rate was similar across conditions: about 98 percent. Local optima were identified at each level. At the data level, 11.66 percent of converged replications were identified with local optima, which were exclusively located in the model with constraints condition. This indicated model without constraints had better estimation performance than the model with constraints. At the model constraints level, 74.75 percent of the converged replications were local optima in the model with constraints condition, whereas it was 0.31 percent for the model without constraints condition. This indicated local optima were much more prevalent while estimating with model constraints. At the starting values level, those in the model with constraints condition converged to different (and thus local) optima about 12 percent of the time when compared to all 10 estimations, and around 2 percent when compared to only five estimations, except for the extremely low starting values condition (78.97 and 76.34 percent respectively). The percentage was much lower in the model without constraints condition, ranging from 0 to 0.72. In conclusion, model constraints had a higher probability for convergence to local optima in general. The worst choice was to use extremely low starting values along with model constraints. As the model constraints were set at the lower limits for main effect and interaction, searching from the nearby boundary made the estimation more unreliable and unpredictable. In other conditions, using different sets of staring values made little impact on the local optima occurrence. Surprisingly, the extremely high starting values without model constraints performed the best, with the fewest local optima, even fewer than the estimations using true parameters as their starting values. In addition, with better item quality and higher proportion of attribute mastery, there were fewer local optima while estimated with constraints. Those variables had little impact while estimating without constraints. Suggestions were provided to practitioners based on the findings.