| dc.description.abstract | Detecting Differential Item Functioning (DIF) is an early step and very critical to investigate any possible bias between groups (e.g., males vs. females). Many early DIF studies only focused on two-group comparison. However, there are many cases where more than two groups exist: Cross-cultural studies are administered in many countries and any simultaneous comparison across countries is often interest to many researchers. Even for the same administered test in a country, ethnicity is another case for multiple groups. As a need, DIF detection studies among multiple groups have increased as well as the number of DIF methods for multiple groups. Even though there is still not enough study, researchers have compared existing multiple groups DIF detection methods by conducting simulation studies for only uniform DIF items. However, multiple groups DIF detection methods including the Generalized Mantel-Haenszel and the Logistic Regression have not been assessed in any simulation study to determine how well these methods control type I error, power and precision for nonuniform DIF items. This dissertation examined the performance of two non-IRT based multi-groups DIF detection methods on the type I error, power and precision rates for both uniform and nonuniform DIF items. Two methods used in the study are the Generalized Mantel-Haenszel (GMH) and the Generalized Logistic Regression (GLR). A simulation study was conducted in addition to a real data analysis. In the simulation study, total number of groups, groups experiencing DIF, the types and the magnitudes of DIF were manipulated factors. These manipulated factors were considered to get an insight for the real data analysis done in advance. Only dichotomously scored data that was generated by using 2PL IRT model was considered. Total number of items was 50 and first 5 items were DIF items. There were 58 total cases and 168 outcomes of interest. 1,000 iterations were used for each case to ensure the accuracy of results. For all cases, type I error was the ratio of falsely detected non-DIF items over all non-DIF items (45 items), power was the ratio of truly detected DIF items over total number of DIF items (5 items), and precision was the ratio of truly detected DIF items over all detected items (the items that were above the detection threshold). Type I error, power and precision rates were calculated as the average of 1,000 iterations for cases. The research questions examined in this study were; (1) In investigating uniform DIF, does the magnitude of DIF affect the performance of the GMH and the GLR under different number of total groups and different groups experiencing DIF? What are the type I error, power, and precision rates of these two methods for these conditions?, (2) In investigating nonuniform DIF, does the type of nonuniform DIF affect the performance of the GMH and the GLR under different number of total groups and different groups experiencing DIF? What are the type I error, power, and precision rates of these two methods for these conditions? The study showed that, for uniform DIF items, the GMH had slightly higher power and precision rates for two group cases. As the magnitude of uniform DIF increased, the power of two methods increased as well for two groups. For nonuniform DIF items with both a and b parameter change, the results of the GMH was still comparable with the GLR, however, for nonuniform DIF items with only a parameter change, the power and the precision rates of the GMH were very low compared to the GLR. In general, when only one group experienced DIF, methods had the lowest power and precision rates and, the highest power and precision rates when it was reference group experienced DIF. For 6 groups, the GHM had higher power rate for uniform DIF and had similar power rates with the GLR for nonuniform DIF with both a and b parameter change. For 12 groups, both methods had the lowest type I error, power and precision rates when it was medium magnitude of uniform DIF for all cases. Overall, the GLR had better precision rates and lower type I error rates for both 6 and 12 groups. The result indicated that even for nonuniform DIF, the GMH was still able to detect DIF items. However, the type I error rate for both methods were usually above the nominal level of 0.05 with the highest value of 0.2 that meant 9 items were falsely detected. The finding of the simulation study with respect to the type I error rate could explain the findings of real data analysis. All items were found to have DIF with both methods in the real data analysis. However, before concluding that the items is biased toward one group with the real data, further investigation is required by experts. When the power is the only concern, both methods should be used since each of them has its advantages for uniform and nonuniform DIF. However, when the precision is main concern, the GLR is better than the GMH for the majority of cases. | |
