Differences in K-12 science standards and Differential Item Functioning (DIF) in NAEP Science Assessments

Abstract

This study examined the role of differences in K-12 science standards across US states on measurement comparability, as indicated by differential item functioning (DIF), on the National Assessment of Educational Progress (NAEP) science assessments at grades 4, 8, and 12. Restricted data sets as offered and maintained by NCES were used in this investigation. Extant data was obtained from Institute for Education Sciences (IES) and includes item-level responses of students along with student characteristics, school district and state identification. The report of Gross et al 2013, which evaluated state science standards against Next Generation Science Standards, was used to categorize states’ science standards as superior or inferior. The Mantel-Haenszel and Logistic regression analysis procedures were used to conduct DIF analysis. The findings evidenced that there is DIF based on differences in state science standards in three (3) test questions out of the 126 test questions asked at 12th grade. However, no DIF based on differences in state science standards have been found on the 4th and 8th grade examination items. In these DIF detected items on the 12th grade exams, the odds of students from superior states responding correctly to these items are 1.537 to 1.664 times higher than that of students from inferior standard states. These items were from physical science and life science content areas and favor students from superior science standard states over students from inferior science standard states. However, no DIF items were detected in earth and space science content area. Additional analyses have shown that the results are robust (i.e., no DIF detected) when controlled for the gender and the race of students. Further, examinee ability was measured at the content level and the results evidence that using this alternative measure of ability had no effect on the main findings. Overall, the findings strongly suggest that differences in state K-12 science standards is associated with the degree to which NAEP science assessments measure what they claim to measure: the true science ability of students and become an important factor when assessing measurement validity.