RELA: An Online-Offline Framework for Rules Enhanced Logs Analysis Based on Large Language Models

Recently, Large Language Models (LLMs) have played a pivotal role in the field of Artificial Intelligence Operations (AIOps), particularly in the analysis of log data. However, the use of open-source LLMs for automated log analysis often falls short of expectations in offline environments in practice. To address these challenges, this paper introduces an innovative Online-Offline framework named RELA to enhance the performance of LLMs in log analysis. In the online phase of our framework, GPT-4 meticulously analyzes logs to determine the presence of anomalies and deduces rules based on these determinations, which are then compiled into a comprehensive rules base. In the offline phase, open-source LLMs leverage this rules base to significantly enhance the effectiveness of log analysis by retrieving the most applicable rules. To validate the effectiveness of our proposed framework, we meticulously annotated a dataset specifically designed for a distinct offline scenario, focusing on two primary tasks: the detection of log anomalies and the explanation of these anomalies. Our experimental results demonstrate that the implementation of the rules base leads to a 3% to 25.3% increase in the accuracy of anomaly detection. Additionally, assessments conducted by domain experts validate the improvements, highlighting a substantial enhancement in the credibility of the explanations provided by LLMs within this framework. These findings not only underscore the practicality but also the efficacy of our approach, establishing a solid foundation for further research and application in AIOps.