Offline Inverse Constrained Reinforcement Learning for Safe-Critical Decision Making in Healthcare

One important application of Reinforcement Learning (RL) is optimizing treatment decisions in healthcare. However, a na¨ıve RL policy can lead to unsafe medical decisions, such as excessive dosages or abrupt treatment changes, often because agents fail to account for common-sense constraints. To mitigate these problems, Constrained Reinforcement Learning (CRL) naturally emerges as a promising approach for safer decision-making by optimizing policies under predefined constraints. To extend CRL to healthcare applications, a fundamental challenge lies in accurately specifying the constraint function for different healthcare scenarios. Recent Inverse Constrained Reinforcement Learning (ICRL) is a promising approach that infers constraints from expert demonstrations. However, ICRL algorithms model Markovian decisions and rely on real-time interactive environments. These settings do not align with the practical requirement of a decision-making system in healthcare, where decisions rely on historical treatment recorded in an offline dataset. To tackle these issues, we propose the Constraint Transformer (CT). Specifically, 1) we utilize a causal attention mechanism to incorporate historical decisions and observations into the constraint modeling, while employing a Non-Markovian layer for weighted constraints to capture critical states. 2) A generative world model is used for exploratory data augmentation, enabling offline RL methods to simulate unsafe decision sequences. In multiple medical scenarios, empirical results demonstrate that CT can capture unsafe states and achieve strategies that approximate lower mortality rates, reducing the occurrence probability of unsafe behaviors.