The Efficacy of Propensity Score Matching in Bias Reduction with Limited Sample Sizes

Abstract

The current literature on propensity score matching is missing imperative information for educational researchers regarding the practical implications of utilizing this method with limited sample sizes. The purpose of this study was to evaluate the effectiveness of propensity score matching when limited by sample sizes of 500,400, 300, and 200 as determined by a reduction in bias using both real and simulated data. Further effort was made to determine the optimal selection of covariates and caliper width with these limited sample sizes. Participants were selected without replacement and matched one-to-one using the nearest neighbor technique in the MatchIt package in the R software program. Contrary to the hypothesis that with reduction in sample size the balance improvement would drop below what is considered effective bias reduction, the reduction in bias was greater than 96.77% for all conditions of sample size and caliper width. A Monte Carlo simulation was created based on the real dataset to assess covariate selection with the same limitations in sample size and a set caliper width of 0.6. For all replications, the mean balance improvement was best for the covariate relationship magnitude strong_none (strong relationship to DV_no relationship to treatment) and worst for the relationship mod_strong (moderate relationship to DV_strong relationship to treatment). Only the covariate relationship strong_none was able to be deemed effective matching for all sample sizes. Findings suggest that propensity score matching can be effective at reducing bias with sample sizes as small as 200 and caliper widths as wide as 0.6. Ideal covariates are those that are strongly related to the outcome variable and only weakly or moderately related to treatment when sample sizes are limited. Keywords: Propensity Score Matching, Sample Size, MatchIt