Does the Timing of Feedback Given to New Coders of the Communication Complexity Scale Affect Coding Reliability Scores?

Abstract

The Communication Complexity Scale (CCS) helps quantify communicative behaviors demonstrated by children and adults with disabilities (Brady et al., 2012 & Brady et al., 2018a). When individuals are asked to learn the CCS coding system, they are led through the CCS online training system comprised of instructional modules and independent coding of gold standard videos. Upon completion of scoring, coders receive their scores in comparison to the gold standard scores as well as feedback depicting the reasoning for the score choice. Achieving high reliability is essential for new coders; thus, the feedback provided should be adequate to assist with refining scoring errors. The current study recruited participants to complete the instructional modules and code three gold standard videos according to the CCS protocol. Eight participants were randomly assigned to either the control group or experimental group. The control group completed all tasks as currently arranged by the CCS online training system. In contrast, the experimental group received more frequent feedback on the treatment condition tasks (i.e., two gold standard videos). Then, the experimental group received the same feedback timing as the control group on the follow-up task (i.e., one gold standard video). After the study was completed, the results were calculated via the Mann Whitney U test to determine if any statistically significant changes were appreciated. No statistically significant changes between the two groups were noted; although, descriptively, members of the experimental group showed higher reliability on some of the treatment conditions and follow-up tasks. Participants in the experimental group indicated a preference for receiving feedback more frequently to assist with scoring accuracy.