Supervised Learning for Log-Based Anomaly Detection

Log-based anomaly detection is crucial in today's world for security, operational efficiency, compliance, and optimizing application performance. A log of a system describes its state during execution and sequentially records events in the system. Anomaly detection in log data involves identifying abnormal behavior or patterns within system runtime information recorded in logs. The current understanding of log-based anomaly detection heavily focuses on advanced algorithms, deep learning techniques, log parsing, and system behavior analysis. Despite the availability of unsupervised models for anomaly detection in system logs, their blind spots and complexity make supervised learning the preferred, more accurate, and transparent option. The BGL log dataset was selected for developing anomaly detection model due to its comprehensive coverage of system logs and its suitability for identifying patterns associated with anomalies in large-scale systems and preprocessing the data by eliminating noise. Log parsing is done to extract information from logs and convert it into fields using data frames and dropping unwanted data. For For feature extraction, the frequency of specific keywords within log messages was tracked, and these keywords were grouped using Block IDs as the primary key for model input. While traditional approaches have primarily relied on unsupervised methods for log-based anomaly detection since log data is complex, recent progress has highlighted the effectiveness of supervised learning techniques in improving accuracy. Random Forest Ensemble learning, known for reducing variance and overfitting, is a preferred choice for model training. The training and testing data are split as 75% and 25% respectively. The machine learning model shows excellent performance with an accuracy of 99%. It attains elevated precision, recall, and F1-scores for each class, demonstrating strong results for class ‘0’ at 1.00 for precision and 0.99 for recall and F1-score.