SUBSCORE REPORTING COMPARISON BETWEEN THE LOG-LINEAR COGNITIVE DIAGNOSTIC MODEL AND THE HABERMAN MODELS

Abstract

Academic assessments are designed and used for a variety of purposes. One of the primary functions of any of these tests is to diagnose the respondent. In all forms of diagnosis, a single or series of assessments are conducted. Based on the outcome of these assessments, a decision can be made as to the condition of the person or object. For this study, the diagnosis is conducted on a person. This can be in a purely academic setting such as a K-12 school, a post-secondary situation such as a university or trade school, or certification and licensure testing such as medical or another professional field. The goal is to determine the level of a latent trait the respondent possesses through a test designed to identify and measure this trait (Rupp, Templin, & Henson, 2010). It is these subscores that can potentially add to the diagnostic ability of a specific test. If a subscore can provide additional information about the level of a latent trait, the diagnostic ability of the test could be increased. Low psychometric qualities such as having low subscore reliability or having high correlations between the subscores in question and the total score of the test are always of concern (Sinharay, Puhan, & Haberman, 2010). This study looks at the theoretical possibility to use Log-Linear Cognitive Diagnostic Models (LCDM) to examine candidate mastery levels of individual attributes (subscores) within a test while keeping reliability and validity of these subscores psychometrically suitable. This simulation study compared reliability and correlation estimates of five Classical Test Theory (CTT) based subscore methods with the LCDM method. Data sets simulated both CTT data and LCDM data. Both types of data were analyzed by the CTT methods and LCDM. One of the CTT methods produced higher reliability estimates and more accurate correlation estimates than the remaining four CTT methods. However, the LCDM produced the highest reliability estimates and most consistent correlation scores across both data sets.