An Analysis of the Concurrent Validity of the Interactive Computer Interview System as Defined by Student Achievement on the Kansas State Assessments

Abstract

This study's primary research question was to determine whether or not the Interactive Computer Interview System displays concurrent validity as defined by student achievement through a criterion-referenced test. The criterion-referenced test used for this analysis was the Kansas State Reading and Math Assessment. In this study, 40 third-, fourth- and fifth-grade teachers throughout the Olathe School District in Olathe, KS were interviewed using the ICIS screening tool. Mean Kansas Assessment scores were then collected for each of the 40 teachers in the subjects of reading and math from 2010, 2011, and 2012. A Z Score calculation and subsequent residual score was determined to adjust for differing test lengths, difficulty, and the effect of socio-economic status as defined by free/reduced lunch percentage for each individual teacher's class. Finally, a correlation analysis was performed between the mean Kansas Assessment Results from 2010-2012 in both reading and math and that same teacher's total weighted average on the ICIS. In addition to a correlation analysis between a teacher's mean score on the Kansas State Assessments over a three year period and their ICIS total weighted average score, an analysis was also performed using an administrator rating system. Each principal that supervises the staff used in this study was asked to give a rating of that teacher's overall teaching ability on a scale of 1-10. These ratings were then compared to both that same teacher's ICIS total weighted average and their three year mean reading and math scores. No significant correlation was found between a teacher's total weighted average on the ICIS and that same teacher's residual reading (p=-0.17816) or residual math (p=-0.16327). There was also no significant correlation found between the administrator's rating and a teacher's total weighted average on the ICIS (p=0.2498). However, there was a statistically significant correlation at the .05 level between an administrator's rating and their residual reading (p=.3718) and at the .01 level on residual math (p=.4309).