Efficient Log Parsing Method Based on Log Locality Features

Log parsing is indispensable for system maintenance, converting unstructured log data into structured formats (log templates) for further log compression and anomaly detection. The effectiveness of log parsing relies on the efficiency of two key processes: template extraction and log matching. Traditional methods, however, suffer from slow pairwise comparisons for template extraction and the tedious, non-scalable sequential approach for template matching. Our research has uncovered two opportunities for optimization based on two log locality characteristics: logs from the same template tend to cluster sequentially, and there is a limited variety of templates used within given timeframes. To exploit these opportunities, we propose the Multi-Logs Template Extraction (MLTE)-Cache framework. MLTE-Cache leverages the MLTE algorithm to enhance the efficiency of template extraction by grouping similar logs and processing them in batch mode. Furthermore, the framework utilizes a cache-assisted proximity matching algorithm to accelerate the log matching procedure. Through comprehensive experiments on open-source datasets, the MLTE-Cache framework has proven highly effective, maintaining a high level of accuracy while delivering a 37% improvement in efficiency.