Empirical Analysis of LLMDPP: Advancing Log Parsing in the LLM Era

In the field of software engineering, the automated analysis of log data is crucial for operations and maintenance teams. This study introduces LLMDPP, a novel log parser that leverages Large Language Models (LLMs) and Determinantal Point Process (DPP) sampling techniques to enhance the efficiency and accuracy of online log parsing. LLMDPP transforms raw log messages into structured log templates through a few-shot learning, thus simplifying the processing and analysis of log data. The study explores the accuracy of LLMDPP in log-parsing tasks and compares the effectiveness of different encoding functions (TF-IDF and Flan-T5-small embedding layer) in DPP sampling. Experimental results show that LLMDPP outperforms traditional methods in both Global Accuracy and Parsing Accuracy, with the semantic information encoding function performing better when the number of samples is low. Furthermore, we simulate an online scenario to evaluate the parsing effectiveness of different sampling methods on unseen log datasets. The results indicate that the DPP sampling method has an advantage in maintaining sample diversity and fairness, which can improve parsing accuracy.