Energy-Based Predictive Root Cause Analysis for Real-Time Anomaly Detection in Big Data Systems

As the scale of data continues to grow exponentially, managing resource allocation and energy consumption in big data systems becomes increasingly complex and critical. Moreover, with big data systems, energy efficiency is more important daily. In cloud environments, it can be the determining factor between reduced costs and lowered environmental damage. This paper presents a deep learning-based framework for accurately predicting instant energy consumption in real-time and detecting anomalies of different sizes in big data clusters. We use SmartMonit to gather task execution and real-time infrastructure data. A Feedforward Neural Network (FNN) predicts energy consumption from CPU utilisation, memory usage, and task profiling research. The system will track any deviation from predicted consumption with root cause analysis (RCA) if there are significant anomalies. We also integrate an Autoencoder to identify straggler tasks and inefficient resource utilisation. Userdefined functions are next applied to examine these anomalies and try to detect the underlying reasons, like distributed data processing, locality of computation exploitation, or resource waste. Given the scale and heterogeneity of big data workloads, the system's ability to dynamically adjust and optimise resource usage is essential for handling complex processing tasks. The experimental results prove that the proposed system effectively enhances resource allocation and decreases wasted energy.