Loading...
Simulating a Glass Escalator: in silico modeling of measurement bias in teacher evaluations
Ralph, Michael
Ralph, Michael
Citations
Altmetric:
Abstract
The United States teaching workforce fails to represent the US population across many subgroup categories, including gender. One mechanism by which the overrepresentation of men among principals relative to the percent of men teaching in classrooms may be reproduced is through biased teaching evaluations that confer an inappropriate advantage to men. I ran simulations to test whether biased teacher evaluators would produce measurable effects in commonly used measures of rater reliability for teacher evaluation systems. I then tested whether relatively simple changes to rater designs would increase the detectability of rater bias in measures of reliability. The results show no single measure of reliability, either from Classical Test Theory or a Rasch Rating Scale Model, consistently detected rater bias. Instead, the most consistent indicator of bias in the study was the direct comparison of subgroup results themselves. These findings demonstrate the potential in real data studies for subgroup differences to result from validity threats. Prior to attributing observed differences to the subgroup members being observed, researchers should rule out validity threats as an explanation for those differences.
Description
Date
2022-01-01
Journal Title
Journal ISSN
Volume Title
Publisher
University of Kansas
Collections
Archive Status
This item contains archived web content.
Files
Ralph_ku_0099D_18691_DATA_1.pdf
Adobe PDF, 2.05 MB
- Embargoed until 2172-05-31
Research Projects
Organizational Units
Journal Issue
Keywords
Educational psychology, gender, QuantCrit, rater bias, simulation, teacher evaluation