Random Permutation Testing Applied to Measurement Invariance with Dichotomous and Likert-type Indicator Variables

Abstract

An important detail that appears to be frequently overlooked in the SEM literature is how modeling data arising from responses on an ordered-categorical scale can in- fluence measurement invariance testing. Typically tests for measurement invariance are conducted by comparing the fit of two nested models with chi-square difference testing. With ordered-categorical data the chi-square difference statistic measuring the discrepancy between two models does not follow a chi-square distribution (Muthén & Muthén, 2015), therefore chi-square difference testing is inappropriate. The popu- lar solution to this problem is to use a scaling correction on the chi-square difference statistic to improve its chi-square approximation and test the resulting value for sta- tistical significance (e.g., Garnaat & Norton, 2010; Randall & Engelhard, 2010). The purpose of the present research was to introduce and evaluate random permutation testing applied to measurement invariance testing with ordered-categorical data. The random permutation test builds a reference distribution from the observed data that is used to calculate a p-value for the observed chi-square difference value. The reference distribution is built by repeatedly shuffling the grouping variable and then saving the chi-square difference between the two models fitted to the resulting data. The present research consisted of two Monte Carlo simulations. The first simulation was designed to determine how many random shuffles of the grouping variable are appropriate. The second simulation was designed to evaluate random permutation testing across a va- riety of conditions in comparison to existing chi-square difference testing methods. Simulation results, an empirical example, and suggestions for the use of the random permutation test are provided.