Treatment effect estimation from observational data in healthcare: study designs, inherent biases, and a propensity score-focused quality assessment scale

Abstract Observational data are widely used to understand treatment-effect relationships, particularly in healthcare, when randomized controlled trials are infeasible. Observational studies, however, are susceptible to biases arising from study design, data limitations, and analytic methods. Identifying and addressing these biases is critical and challenging in observational research. This paper elaborates on the sources of such biases; reviews commonly used observational study designs and propensity score (PS)-based methods; and examines existing tools for assessing the quality of evidence from causal studies under the counterfactual inference paradigm. We argue that, although PS-based methods are widely used, existing quality assessment tools place insufficient emphasis on data quality and inference validity and may overlook sources of bias specific to PS-based analyses. To address this gap, we introduce the PSM-Qual Score, a quality assessment tool that evaluates biases related to defining and identifying comparison groups, de-confounding, and interpretability. We also developed an implementation Guideline to support consistent application of the tool. Two independent assessors evaluated inter-rater reliability across 127 published studies using the PSM-Qual Score and Guideline. Agreement ranged from 82.0% to 99.2%, with Cohen’s kappa values from 0.377 to 0.922 and weighted kappa values from 0.382 to 0.958, with most items demonstrating substantial to almost-perfect agreement. The PSM-Qual Score and Guideline support researchers in the exact and computational sciences in using observational causal inference methods appropriately, contributing to data-based healthcare research.