Developing a Taxonomy for Advanced Log Parsing Techniques

Logs are widely used in various software engineering applications, including debugging, program comprehension, failure prediction, and anomaly detection. Despite their value, the unstructured nature of logs complicates the extraction of meaningful insights. In response, various log parsing techniques leveraging methods like machine learning and pattern recognition have been developed. Nevertheless, existing parsers frequently fail to achieve consistent accuracy, especially when handling complex log formats. To address this challenge, we conduct a comprehensive study to understand the characteristics of log events that lead to parsing errors. Using 16 different log datasets and 8 log parsers, we apply open coding techniques to derive a taxonomy of log event characteristics that contribute to parsing errors. We also examine how different log parsers are impacted by each category in the taxonomy. The resulting taxonomy not only provides insights into the complexity of parsing log data but can also guide the development of advanced parsing tools capable of handling the unique characteristics of diverse log formats.