A DEEP LEARNING APPROACH FOR DETECTING ANOMALIES IN DISTRIBUTED SYSTEM LOGS

This paper presents a deep learning-based approach for detecting anomalies in distributed system logs, addressing the challenges posed by the increasing complexity and volume of log data generated in modern computing environments. Distributed systems, characterized by their decentralized architecture, provide enhanced scalability and fault tolerance, yet they also complicate monitoring and diagnostics due to the sheer amount of log data produced. Traditional anomaly detection methods, including statistical and rule-based approaches, often struggle to keep pace with the dynamic nature of log data and the high dimensionality of the information contained within. In response to these limitations, we propose the use of Long Short-Term Memory networks, a type of recurrent neural network adept at capturing temporal dependencies in sequential data, to effectively identify anomalies in log entries. Our methodology involves systematic data collection from diverse sources, rigorous data preprocessing, and the application of deep learning techniques to develop a robust anomaly detection model. The experimental results demonstrate that our LSTM-based approach significantly outperforms traditional methods, achieving high accuracy, precision, and recall rates in identifying both known and unknown anomalies. This research contributes to the field by providing a scalable and effective solution for log analysis in distributed systems, ultimately enhancing system reliability and security. Future work will explore the integration of additional deep learning architectures and dynamic thresholding techniques to further improve anomaly detection capabilities.