Distributed System Log Anomaly Detection Method Based on LSTM Networks and Process State Inspection

ABSTRACT Runtime anomaly detection in software plays a critically important role in the management of many large‐scale distributed software systems. Among numerous anomaly detection algorithms, log‐based detection methods are very popular and efficient. Log systems, existing in virtually all software, including large‐scale distributed systems, typically record extensive details about system operations, user activities, and software exceptions. These logs serve as a vital channel for researchers to understand system status and performance and to analyze anomalies. However, traditional anomaly detection approaches, which rely on manual log queries by researchers, are time‐consuming and inefficient. Moreover, as the scale and complexity of distributed systems increase, the exponential growth in the volume of logs renders these methods impractical. We propose an anomaly detection algorithm based on deep neural network models that utilize system logs and incorporate process state checks to detect anomalies, proving particularly effective in simulating distributed denial of service (DDoS) attacks. The model employs long short‐term memory (LSTM) networks to model the encoded event templates as natural language sequences, automatically learning log patterns during normal execution. Furthermore, we model the performance of core components in distributed systems (e.g., Hadoop HDFS DataNode) to explore their state space. Compared with two mainstream log‐based anomaly detection algorithms, our model achieves optimal performance on datasets simulating DDoS anomalies.