The Impact of the Item Types and Number of Solution Steps of Multiple-Choice Items on Item Difficulty and Discrimination and Test Reliability

Abstract

This study examined two multiple choice item-writing guidelines addressed by Haladyna, Downing, and Rodriguez (2002). One is related to using the "None of the Above (NOTA)" option, the other is about the plausible number of options for a multiple-choice item (MCI). These two guidelines were empirically tested using one-step and multi-step problems to identify their impact on item characteristics (item difficulty and item discrimination) and test characteristics (test reliability). Three forms with MCIs were generated and administered to approximately 1500 7th and 8th grade students in the United States and Turkey. Bi-factor Item Response Theory (IRT) was applied to assess dimensionality related to the number of solution steps of items. Multiple regression models were employed to determine the degree of impact these item-writing guidelines had on item and test characteristics for MCIs with one step solution (MCI with one-step solution) and those with more than one step solutions (MCIs with multi-step solution). The results show that item characteristics do not change significantly across the conventional MCIs with four options, MCIs with three options, and MCIs with NOTA option. The interaction between solution steps and the three MCI types had no significant impact on item characteristics. For the test with MCIs with a one-step solution, the findings demonstrate that four options are significantly more reliable than the NOTA options and not statistically different from three options. For the test with MCIs with multi-step solutions, four options are not statistically different from three and NOTA options. Compared to MCIs with four options, the results support that MCIs with NOTA options are preferable for MCIs with multi-step solutions while three options are desirable for both MCIs with one-step solutions and multi-step solutions.