Reproducible Evaluation Framework for Sepsis Treatment Strategies Learned via Offline Reinforcement Learning

Sepsis is a life-threatening condition caused by the body's response to an infection, leading to widespread inflammation and potential organ failure. If untreated, it can progress to septic shock, characterized by a dangerous drop in blood pressure, severe organ damage, and death. Treatments include intravenous fluids and vasopressors, though optimal strategies are debated. Prior work has explored using offline Reinforcement Learning (RL) to find optimal treatment strategies for sepsis with the intent of being integrated into a Decision Support System (DSS). Each such work uses RL but varied definitions of sepsis state and patient inclusion-exclusion criteria to design its policy learning strategies, making comparison of results difficult. This paper presents the first consistent evaluation of several RL reward functions using a single consistent patient state and cohort criteria confirmed by clinicians as generally representative and a standardized evaluation framework. We find that none of the functions outperform individual physician policies. We highlight the need for better quantification of uncertainty in decision-making for RL in critical care. We also release a library of these functions to the open-source community for reproducible validation and improvement (https://github.com/IpreuettUW/sepsis_treatment_strategy_eval_framework/).