Beyond Propensity Scores: A Tolerance-Based Exact Matching Approach to Impact Evaluation

Impact evaluation relies on credible counterfactual construction, and propensity score matching has long served as the default tool for this task. Recent criticism, most notably the "propensity score paradox" identified by King and Nielsen (2019), has questioned whether propensity score matching reliably improves covariate balance relative to simpler alternatives. This paper develops and applies a tolerance-based exact matching approach that departs from coarsened exact matching at one specific point: categorical variables are matched exactly, while numerical variables are matched within fixed tolerance bands (typically five to ten percent) rather than fixed-width bins. The approach is combined with the Callaway and Sant'Anna (2021) difference-in-differences estimator to allow for staggered treatment timing. Using a hypothetical, synthetic dataset constructed to reflect the structure of administrative firm-level data, three matching specifications are compared against propensity score matching. The tolerance-based approach achieves substantially tighter covariate balance (standardized mean differences between 0.00 and 0.08) but retains a smaller matched sample (n=71), while propensity score matching achieves looser balance (0.22-0.28) with a larger sample (n=386). The paper does not claim that one method dominates the other; rather, it argues that the trade-off between balance and sample size should be made explicit and auditable, and that the tolerance-based approach offers a transparent alternative for settings where the propensity score's statistical opacity is a practical concern.