Using Noncompensatory Models in Cognitive Diagnostic Mathematics Assessments: An Evaluation Based on Empirical Data

Abstract

The present study evaluates the performance of four noncompensatory cognitive diagnostic models -- AHM, DINA, Fusion, and Bayesian Networks -- using both formative and large-scale mathematics assessments (Fraction dataset, TIMSS dataset, and Slope dataset). The author describes the four models in terms of their parameter estimation results and global model-fit conditions, respectively. With regard to the latter, the posterior predictive model checking technique with the number-correct score serves as the discrepancy measure. The author describes model reliability within the Cognitive Diagnostic Modeling framework via the correct classification rate and test-retest consistency rate. Moreover, the author contrasts the DINA, Fusion, and Bayesian Networks model in terms of the discrimination of examinees at different mastery levels on each of the skills indicated by the Q-matrix. Ultimately, the results of examinee classification on individual bases for each of the model pairs are presented. Results of this study suggest: (1) the Attribute Hierarchy Model is better at classifying students when theoretical predictions about attribute dependencies are specified a priori. (2) Even though the DINA model is considered as a simple CDM, it can provide a greater degree of information regarding student classification for certain dataset. (3) The Fusion Model can better fit on the Fraction and TIMSS dataset, and the computation cost was lower than anticipated. (4) While the Bayesian Networks approach proved a flexible technique, it was superior with respect to fit and model interpretation on the Slope dataset and Fraction dataset within this study.