Contextual Learning with Online Convex Optimization: Theory and Application to Medical Decision-Making

Optimizing the treatment regimen is a fundamental medical decision-making problem. This can be thought of as a two-dimensional decision-making problem with a nested structure because it involves determining both the optimal medication and its optimal dose. Identifying the most effective medication for an individual often poses considerable difficulty, and even when a suitable medication is ascertained, dosing it optimally remains a significant challenge. Making these two nested decisions necessitates the adaptive learning of a personalized disease progression control model. To address this problem, we propose a novel contextual multiarmed bandit model under a two-dimensional control with a nested structure. For this model, we develop a new joint contextual learning and optimization algorithm, termed the stochastic subgradient descent atop contextual multiarmed bandit (SGD-MAB) algorithm. It sequentially selects for a patient (i) the best medication based on their contextual information and (ii) the corresponding dose optimized over the prior history of those patients who received the same medication. We prove that it admits a sublinear regret, which is tight up to a logarithmic factor. Our regret analysis leverages the strengths of both contextual bandit approaches and online convex optimization techniques in a seamless fashion. We substantiate the practicality of SGD-MAB using clinical data on patients with hypertension and heightened cardiovascular risks. Our analysis indicates that SGD-MAB has the potential to surpass current practices. We benchmark several policies to show the advantages of our approach and offer critical insights. Our framework holds promise for various applications beyond healthcare that require nested decision-making. This paper was accepted by J. George Shanthikumar, data science. Funding: This work was supported by the National Science Foundation (CMMI-1548201, CMMI-1634505) and the National Eye Institute (NIH Grant R01EY026641). Supplemental Material: The online appendix and data files are available at https://doi.org/10.1287/mnsc.2019.03211 .