The Effects of Different Scoring Methodologies on Item and Test Characteristics of Technology-Enhanced Items

Abstract

Technology-enhanced (TE) item types have recently gained attention from educational test developers as a way to test constructs with higher fidelity. However, most research has focused on developing new TE item types, and less on researching best practices for scoring these new item types. The purpose of this study was to analyze the effect of adjusting scoring strategies of TE items on item and test characteristics. Descriptive statistics as well as tests of statistical significance were reported when appropriate. Additionally, figures representing the differences in test information and fit across forms were created to help show consistency in scoring effects. Results were consistent with prior research into differences between dichotomous and polytomous scoring strategies. Results indicate that the two best strategies for scoring TE items are partial-credit scoring and testlet response theory. The worst approach to scoring TE items is to score them as correct-only. Results of this study add to the research literature, as well as provides a practical guide to test developers when deciding which scoring strategy to use with new TE item development.