Reinforcement Learning-Based Self-Healing Framework in IoT Sensor Networks

The fast increase of devices in Internet of Things (IoT) networks, which is projected to grow to over 21 billion devices by 2025, will require more sophisticated management paradigms, as current rule-based and reactive frameworks cannot support networks of this scale. Rule-based systems are capable of processing only of predefined failure patterns and when it comes to complex situations like simultaneous multiple failures and cascading failures, they cannot work. The proposed paper suggests a self-healing framework of reinforcement learning (RL) with respect to an IoT sensor network. The original innovation of the framework is that it brings the MAPE-K cycle (Monitor-Analyze-Plan-Execute over Knowledge) framework, the fundamental reference model of autonomous computing, to which a learning element is added to form the MAPE-K+L model. This aspect provides the system to be able to enhance its policy gradually through learning on the past failures. The proposed framework has been tested in a custom Python/Gymnasium simulation framework, with six failure modes (single node failure, sensor drift, gateway failure, concurrent multiple failure, network congestion, cascading failure) in cluster topology networks, using 50 to 500 nodes. The Q-Learning and Deep Q-Network (DQN) agents were fully contrasted with random (RND) and rule-based (RB) baselines. The Q-Learning agent in the multiple failure scenario decreased the mean recovery time (MTTR) by 51.9 and 32.8 percent relative to random selection and rule-based approach respectively (p<0.001, Cohen d=1.424). The DQN agent had the best cumulative reward and the most stable performance in the cascading failure case; scalability experiments proved that DQN can work with a stable performance even in the 500-node networks.