Strategies to deal with ordinal missing data for measurement invariance testing and specification searches – A comparison of commonly used methods

Abstract

Measurement equivalent/invariance is a key concept in psychological testing. Failing to correctly identify non-invariant items can lead biased group comparisons and biased selections. The methodological literature on measurement equivalent/invariance (ME/I) and specification searches in structural equation modeling (SEM) usually consider only complete data. In practice, ME/I tests are often done on Likert scales which involve ordinal variables. Missing data on ordinal variables can be problematic in ME/I tests based on the chi-square statistic ( ) and modification indices. To deal with missing ordinal data, a recommended strategy is to combine multiple imputation with weighted least squares estimation methods. However, both statistic and modification indices are not available with this strategy. Consequently, researchers have to adopt “suboptimal” methods: 1) use full information maximum likelihood (FIML) by treating ordinal data as normally distributed continuous data; 2) use robust FIML by treating ordinal data as non-normally distributed continuous data; and, 3) use weighted least squares (WLSMV) estimators with suboptimal missing data handling techniques, such as pairwise deletion. Previous studies have found that any of the strategies may bias the point estimates and statistics in SEM. Yet, there has been no systematic comparison of the suboptimal strategies, especially in the context of ME/I tests or chi-square difference tests (Δ tests). Thus, the goals of my dissertation are to investigate the relative performance of these commonly used suboptimal strategies on the Δ tests and modification indices in ME/I testing with ordinal missing data. Two simulation studies were conducted. Study 1 aimed to compare the three strategies in terms of the accuracy and efficiency of parameter estimates as well as the type I error rate and power of Δ tests. Study 2 aimed to examine the relative performance of the strategies on specification search. I investigated three backward specification search methods based on the largest modification index using the three suboptimal methods described above and compared it to a recently proposed forward specification search method based on confidence intervals (CI approach), which can be implemented in the “optimal” approach of WLSMV using multiple imputations. The first simulation study showed that when the target data set contains a substantive amount of ordinal missing data, using the Δ tests and modification indices obtained from WLSMV with pairwise deletion lead to a substantive inflation of type I error rates. In contrast, the Δ tests and modification indices obtained from FIML approaches had a better ability to control the type error with sufficient power to test measurement invariance under most conditions. However, parameter estimates were biased for the FIML approaches. In the second simulation study, FIML based modification indices could identify more effectively the correct invariant factor loadings than the modification indices from the WLSMV estimator using pairwise deletion or the CI approach from the WLSMV estimator with multiple imputations. However, all search methods showed an inflated type I error at the model level because none of the methods could effectively locate non-invariant thresholds. Future directions of the ordinal missing data in invariance testing are discussed and practical suggestions for empirical researchers are provided.