LogProb: Online Parsing Evolving Logs With Complex Parameters

To cope with the massive volume of log messages in complex systems, effective and accurate log parsers are crucial for system maintenance. However, existing log parsers have accuracy issues, particularly when handling evolving logs with complex structures. To address this, this paper proposes LogProb, an online log parsing method that incorporates a token state prediction component and a search tree-based template extraction component. The token state prediction component determines whether each token in a log message is static (template word) or dynamic (parameter value). The token sequence and predicted token states are parsed by the template extraction component to generate event templates. The extraction process includes two stages: identifying candidate templates with predicted static tokens, and using a fast matching algorithm to determine the final template. Experiments on 16 benchmark datasets show that LogProb outperforms state-of-the-art methods in accuracy while maintaining comparable efficiency.