Loading...
A STUDY TO IDENTIFY FACTORS OF RATER COGNITION FROM DIFFERENT RATER PERSPECTIVES DURING PERFORMANCE-BASED ASSESSMENTS
Lewis, Heather A
Lewis, Heather A
Citations
Altmetric:
Abstract
Performance-based assessments (PBA) are used in healthcare education and for the evaluation of performance in professional practice. In nursing, PBAs are used with observation to assess competency of students and professionals. Rating using a PBA is highly complex, and the literature shows high levels of rater variance in rater scores and the ability of a rater to accurately assess and rate a performance. The field of rater cognition studies the variety of cognitive factors influencing a rater’s ability to accurately observe, process, and integrate a judgement about the performance into a scale. The purpose of this mixed-methods study was to explore rater cognition from three perspectives of raters who watched and rated the same performance of nurse practitioner students participating in a telehealth exam with a standardized patient. The student, standardized patient, and advanced practice faculty evaluated the student from their rating perspective through evaluation of the PBA scores, reflection questions, and interviews. A framework developed by Gauthier et al. (2016) was used to guide the exploration of rater cognition. Five student performances were rated by a total of five students, three faculty, and three standardized patients. Each triad of raters completed the PBA, reflection questions, and semi-structured interviews. Quantitative analysis of the PBA scores for each rater triad (scoring the same student performance) was completed using pairwise comparisons, triad comparisons, and individual items. The most disagreement occurred with the student and faculty pair. One of the triads had a higher number of disagreements relative to the other triads, and two PBA items had disagreement with at least three different triads. Qualitative analysis was performed on the reflection questions and interview questions. Keyword analysis was completed on the reflection questions and identified 18 keywords. Content analysis of the interviews found five themes with fifteen categories that addressed the research questions. The themes identified were: The Measuring Sticks of Competence, Nothing is Perfect: Rater Views of the PBA, the Positive and Negative Effects of Raters Assessing Performances Sequentially, Prior Exposure to the PBA and Scenario is Essential to Understanding Expectations and Scoring, and Additional Factors that Influence Rating. The cognitions of the raters in this study were identified using reflection questions and interviews, providing insight to the similar and unique cognitions of the different rater types. Some cognitions were noted across rater groups, indicating common cognitions. Others were only identified in a singular group, indicating those cognitions may be more closely associated with the specific role. Understanding similar and different cognitions per group can allow targeted interventions to address the cognitions and limit their effect on PBA scoring. PBA score analysis identified potential items for review by the school of nursing and differences in scoring across rater groups. Further research is needed to compare triad data on a larger scale, complete analysis of PBA items, and continue to identify and address rater cognitions. Keywords: performance-based assessment, PBA, nursing education, rater cognition
Description
Date
2021-01-01
Journal Title
Journal ISSN
Volume Title
Publisher
University of Kansas
Collections
Archive Status
This item contains archived web content.
Files
Lewis_ku_0099D_18090_DATA_1.pdf
Adobe PDF, 1004.07 KB
- Embargoed until 2171-05-31
Research Projects
Organizational Units
Journal Issue
Keywords
Nursing, Health education, Education, nursing education, PBA, performance-based assessments, rater cognition, student assessment