Technology Enabled Assessments: An Investigation of Scoring Models for Scaffolded Tasks

Abstract

While significant progress has been made in recent years on technology enabled assessments (TEAs), including assessment systems that incorporate scaffolding into the assessment process, there is a dearth of research regarding psychometric scoring models that can be used to fully capture students' knowledge, skills and abilities as measured by TEAs. This investigation provides a comparison of seven scoring models applied to an operational assessment system that incorporates scaffolding into the assessment process and evaluates student ability estimates derived from those models from a validity perspective. A sequential procedure for fitting and evaluating increasingly complex models was conducted. Specifically, a baseline model that did not account for any scaffolding features in the assessment system was established and compared to three additional models that each accounted for scaffolding features using a dichotomous, a polytomous and a testlet model approach. Models were compared and evaluated against several criteria including model convergence, the amount of information each model provided and the statistical relationships between scaled scores and a criterion measure of student ability. Based on these criteria, the dichotomous model that accounted for all of the scaffold items but ignored local dependence was determined to be the optimal scoring model for the assessment system used in this study. However, if the violation against the local independence assumption is deemed unacceptable, it was also concluded that the polytomous model for scoring these assessments is a worthwhile and viable alternative. In any case, the scoring models that accounted for the scaffolding features in the assessment system were determined to be better overall models than the baseline model that did not account for these features. It was also determined that the testlet model approach was not a practical or useful scoring option for this assessment system. Given the purpose of the assessment system used in this study, which is a formative tool that also provides instructional opportunities to students during the assessment process, the advantages of applying any of these scoring models from a measurement perspective may not justify the practical disadvantages. For instance, a basic percent correct score may be completely dependent on the specific items that a student took but it is relatively simple to understand and compute. On the other hand, scaled scores from these scoring models are independent of the items from which they were calibrated from, but ability estimates are more complex to understand and derive. As the assessment system used in this study is a low stakes environment that is mostly geared towards learning, the benefits of the scoring models presented in this study need to be weighed against the practical constraints within an operational context with respect to time, cost and resources.