Design and Implementation of Online Log Parsing Method Based on ALBERT and K-Means ++ Clustering

Log resolution is a method to convert unstructured or semi-structured log data into structured log templates. It is often used as an intermediate link to serve downstream tasks such as log anomaly detection. In view of the log management statistics and abnormal information search requirements of the operation monitoring of the ground system of Fengyun 4B satellite, it is difficult for existing methods to analyze the logs of complex structure with high accuracy. This paper takes the logs of planning and scheduling software and equipment monitoring software as the research object, and proposes an analysis method that can adapt to the logs of complex structure and generate a log template to provide an overview of automatic online log analysis. In this method, a preprocessing process is designed to improve the generalization, and the ALBERT model, PCA dimensionality reduction method and K-Means ++ clustering method are used to improve the efficiency while ensuring the accuracy. The method is applied to the open source HDFS data set and the software log data set of Fengyun 4B satellite ground system, and both the accurate analyses is more than 99%. The analysis results of this method and existing analytical methods applied to the log data set of monitoring software show that the accuracy of this analytical method is improved to 15%, and the running time is shortened by more than 70%, which is the best analytical method. Therefore, this method is an effective method for analyzing complex engineering logs with high accuracy.