Model-Based Q-Learning with Monotone Policies for Personalized Management of Hypertension

Hypertension is a crucial controllable risk factor of atherosclerotic cardiovascular disease, a leading cause of death in the United States. While traditional analytic techniques may capture the complexities of hypertension treatment planning, they generally provide unintuitive treatment recommendations. This paper aims to advance the acceptance of analytic techniques in clinical practice by presenting a method to obtain interpretable treatment plans. To this end, we introduce the monotone Q-learning algorithm, which guarantees policies are nondecreasing on patients' health severity by limiting the exploration of treatment choices and solving simple integer programs. We represent a set of clinically representative patient profiles through Markov decision process models and compare the performance of our approximately optimal monotone policies with the optimal policy, optimal monotone policy, and current clinical guidelines. The approximately optimal monotone policies outperform the current clinical guidelines while displaying small losses in quality-adjusted life years compared to the optimal policy.