RS-DRL: Managing Uncertainty in Self-Adaptive Systems Based on a Novel Continuous Deep Reinforcement Learning Method

Designing software systems within uncertain and evolving operational contexts is challenging, as pervasive uncertainty can compromise system objectives. Self-adaptive systems address this by dynamically adjusting configurations when objectives are at risk. However, exhaustive exploration of large configuration spaces is time-consuming and computationally expensive. Existing machine learning, search-based, and reinforcement learning approaches often degrade under changing conditions. To overcome these limitations, this paper introduces an architecture based on the MAPE-K loop integrated with a Deep Reinforcement Learning (DRL) module enhanced by a novel reward-shaping mechanism. The proposed Reward-Shaped Deep Reinforcement Learning (RS-DRL) method reshapes rewards during experience replay, improving generalization, convergence, and adaptability across dynamic environments. Experiments on IoT case studies (DeltaIoTv1/v2) show that RS-DRL achieves an asymptotic multi-objective reward of \(0.9905\pm 0.0268\) compared to \(0.4559\pm 0.2233\) for the \(\epsilon\) -greedy DQN approach, and near-optimal per-objective asymptotic results (e.g., packet loss \(0.9850\pm 0.0071\) , latency \(0.9964\pm 0.0107\) ). Under the TT goal, RS-DRL reduces packet loss by up to \(22.23\%\) and latency by up to \(39.07\%\) compared to the best competing method ( DLASER+ ), while also consistently outperforming the exhaustive-search Reference baseline that analyzes the full adaptation space. All comparisons are based on 30 independent runs, with \(25\) of \(28\) baseline comparisons statistically significant (many with \(p<0.001\) ). These results demonstrate that RS-DRL offers a robust, efficient, and adaptive optimization strategy for self-adaptive systems operating under uncertainty and dynamic conditions.