Treatment effect estimation by comparing observed and predicted outcomes: conditions for valid inference and practical illustration

BACKGROUND: One non-randomized approach to estimate the average effect of a newly introduced treatment is to compare observed outcomes under the new treatment with predicted outcomes under the standard treatment. These counterfactual predictions are made using a model developed before the new treatment was introduced, using patient characteristics and individualized treatment information. Although the approach has been intuitively applied (e.g., as model-based clinical evaluation in radiotherapy) and can be recognized as a specific case of standardization, a method of virtual controls or a g-method, the theory and conditions required for unbiased treatment effect estimation have not been formally described. The objective of this paper is to formalize the approach and clarify these conditions. METHODS: We formalize the approach within the potential outcomes framework for causal inference. We explain the methodology, its necessary conditions, and approaches for assessing their validity. These conditions are furthermore illustrated through a case study from radiotherapy, estimating the benefit of proton therapy compared to photon therapy on dysphagia in patients with head and neck cancer. RESULTS: We describe a set of five sufficient conditions, including examples of violations: transportability, ignorability of treatment assignment, consistency, positivity, and correct model specification. While these conditions are largely untestable, we describe how empirical evidence, such as comparing predicted and observed outcomes in related samples, can increase confidence in their plausibility. CONCLUSION: When the prediction model predicts well in relevant (sub)populations, the approach can yield unbiased treatment effect estimates. However, there are many possible sources of bias. Therefore, we recommend systematic consideration of all required conditions, informed by domain expertise, and the use of empirical evidence whenever possible to support their plausibility.