Adaptive heartbeat regulation using double deep reinforcement learning in a Markov decision process framework

The erratic nature of cardiac rhythms can precipitate a multitude of pathologies. Consequently, the endeavor to achieve stabilization of the human heartbeat has garnered significant scholarly interest in recent years. In this context, an adaptive nonlinear disturbance compensator (ANDC) strategy has been meticulously developed to ensure the stabilization of cardiac activity. Moreover, a double deep reinforcement learning (DDRL) algorithm has been employed to adaptively calibrate the tunable coefficients of the ANDC controller. To facilitate this, as well as to replicate authentic environmental conditions, a dynamic model of the heart has been constructed utilizing the framework of the Markov Decision Process (MDP). The proposed methodology functions in a closed-loop configuration, wherein the ANDC controller guarantees both stability and disturbance mitigation, while the DDRL agent persistently refines control parameters in accordance with the observed state of the system. Two categories of input signals, namely normal signals and MDP-based stochastic signals, are administered to assess the system's efficacy under both standard and uncertain conditions. Furthermore, the influence of pathological neural activity is emulated through the introduction of external signals characterized by eight discrete frequency components. Quantitative assessments employing metrics such as peak amplitude, signal energy, and zero-crossing rate are performed for each state of the cardiovascular model. The findings substantiate that the ANDC-DDRL strategy effectively stabilizes cardiac rhythms across diverse conditions, surpassing the performance of conventional baseline methods.