Evaluation of Attribute Structure for a Dynamic Assessment Using the Loglinear Cognitive Diagnosis Model and Bayesian Networks

Abstract

In the field of assessment, the construction of a test is critical in matters of pinpointing the use and purpose of the test, the models to be used to generate results, and the inferences that can be made from the test results. Although an attribute map is not necessary to construct a good assessment, a series of well-delineated sets of attributes and a set of well-developed items written to the attributes are essential. As the practitioners of multidimensional test and diagnostic classification models (DCMs) grow, it is important to examine the underlying structural models of attributes within a test. The dissertation seeks to examine the possible structural models of the attributes, using both real data, the Diagnosing Teachers’ Multiplicative Reasoning assessment (Bradshaw et al., 2014) and simulated data in the framework of Loglinear Cognitive Diagnosis Model (LCDM) and Bayesian Networks. Additionally, this research explores the methodology for possible attribute structures that maximizes the impact of the map structure to the implementation and development of a diagnostic assessment. Results from the analysis indicate that the selection of attribute structure can have some implications for attribute parameter estimates and student mastery classifications. The findings also show that sample size and test length have more impact on item level parameter estimates. In addition, the results demonstrated that LCDM integrated with Bayesian Networks is a feasible methodology to detect attribute hierarchy, and thus is a practical choice for multidimensional test scoring.