Evaluating and testing for actionable treatment effect heterogeneity

Developing tools for estimating heterogeneous treatment effects (HTEs) and individualized treatment effects has been an area of active research in recent years. While these tools have proven to be useful in many contexts, a concern when deploying such methods is the degree to which incorporating HTE into a prediction model provides an advantage over predictive methods that do not allow for heterogeneity in treatment effect across individuals. To address this concern, we propose a procedure that evaluates the extent to which an HTE model provides a predictive advantage. Specifically, our procedure targets the gain in predictive performance from using a flexible predictive model incorporating HTE versus an alternative partial linear model which is similar to the HTE-utilizing model except that it is constrained to not allow heterogeneity in treatment effect. By drawing upon recent work in using nested cross-validation techniques for prediction error inference, we generate confidence intervals for this measure of gain in predictive performance, which allows one to directly calculate the level at which one is confident of a substantial HTE-modeling gain in prediction—a quantity that we refer to as the predictive HTE <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML" display="inline" overflow="scroll"> <mml:mi>p</mml:mi> </mml:math> -value. Our procedure is generic and can be directly used to assess the benefit of modeling HTE for any method that incorporates treatment effect heterogeneity.