Offline Reinforcement Learning for Bioprocess Optimization with Historical Data

The development of optimal control strategies for bioprocesses has become essential in pharmaceutical and industrial applications due to the growing demand for sustainable bioproducts. Traditional model-based methods rely heavily on the accuracy of the system model, requiring frequent recalibration and experimentation to sustain performance. In contrast, advanced model-free control techniques, such as Reinforcement Learning (RL), are widely researched. However, training RL controllers online is constrained by the need for extensive online interactions with the biosystem environment, which can be costly and present safety risks. To overcome these limitations, we propose leveraging offline Reinforcement Learning algorithms to train control agents using historical data collected from previous bioprocess operations. These agents can subsequently be fine-tuned, improving current control strategies by utilizing past data without extensive real-time interactions with the system. The Effectiveness of offline RL for policy training was demonstrated through an in-silico semi-batch bioprocess case study, where it achieved superior performance compared to alternative machine learning methods. Additionally, the proposed fine-tuning approach successfully transitioned the offline RL-trained policy into the online operational setting, highlighting the practical advantages of combining offline training with targeted online adaptation to improve real-time performance.