Artifact for "Demonstration-Free: Towards More Practical Log Parsing with Large Language Models'"

This is the artifact for reuse and reproduction of experimental results presented in our ASE'24 paper titled "Demonstration-Free: Towards More Practical Log Parsing with Large Language Models". SETUP Install LogBatcher Install LogBatcher from PyPI: pip install logbatcher OR, Install LogBatcher from source: pip install -e . Set your API Key in config.json To run with docker Download the pre-installed docker image logbatcher.tar , and run the following command: docker load -i logbatcher.tar docker images docker build -t logbatcher . docker run -it logbatcher USAGE Data format LogBatcher mainly takes a raw log file (in plain text format) as input and outputs the parsed log file (in CSV format). A raw log file is a log file with each line representing a complete log. Following the data format from LOGPAI, the data can also be a structured log file. A structured log file is a CSV file that includes at least the LineID and Content columns for parsing, with optional EventID and EventTemplate columns for evaluation. Usage example We provide a usage example for more convenient reuse, which is presented as follows. The usage example can be found in the file demo.py. The example provides a test on a specific dataset Apache from LOGPAI. If you want to evaluate LogBatcher on your own dataset, please replace the arguments file_name and dataset_format with your own raw log file path to load log data and the corresponding dataset format to extract the contents. Run python demo.py and find the results in the outputs/parser/test folder. python demo.py Parsing 2000 logs in dataset Apache... 100%|██████████████████████████████████| 2000/2000 [00:04<00:00, 420.55log/s] parsing time: 4.756490230560303 idetified templates: 6 To evaluate the output of the usage example: cd evaluation && python logbatcher_eval.py --config test --dataset Apache Calculating Edit Distance.... 100%|███████████████████████████████████████████████████████████| 2000/2000 [00:00<00:00, 4029110.47it/s] Normalized_Edit_distance (NED): 1.0000, ED: 0.0000, Grouping Accuracy calculation done. [Time taken: 0.002] Start compute grouping accuracy 100%|███████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 2084.64it/s] Grouping_Accuracy (GA): 1.0000, FGA: 1.0000, Grouping Accuracy calculation done. [Time taken: 0.006] Parsing_Accuracy (PA): 1.0000 Parsing Accuracy calculation done. [Time taken: 0.001] 100%|███████████████████████████████████████████████████████████| 6/6 [00:00<00:00, 10677.06it/s] PTA: 1.0000, RTA: 1.0000 FTA: 1.0000 Identify : 6, Groundtruth : 6 Template-level accuracy calculation done. [Time taken: 0.003] Benchmarking To benchmark on all datasets in loghub-2k or loghub-2.0, you can run the following commands: python benchmark.py --data_type [DATATYPE] --model [MODEL] --batch_size [BATCHSIZE] --chunk_size [CHUNKSIZE] --sampling_method [SAMPLINGMETHOD] The description of the arguments can be found in benchmark.py or below: --data_type Datasets type, Options: ['2k', 'full'], default: '2k'. --model the Large Lauguage model used in LogBatcher, default: 'gpt-3.5-turbo-0125'. --batch_size size of a batch query, default: 10. --chunk_size size of a log chunk, default: 2000. --clustering_method clustering method used in the partitioning stage, Options: ['dbscan', 'meanshift', 'hierarchical'], default: 'dbscan'. --sampling_method sampling method used in the batching stage, Options: ['dpp', 'similar', 'random'], default: 'dpp'. To evaluate, run the following command cd evaluation && python logbatcher_eval.py --config logbatcher_2k The description of the arguments: --config The folder name of the outputs, Options: ['test', 'logbatcher_2k', 'logbatcher_full'] --data_type Datasets type, Options: ['2k', 'full'], default: '2k' --dataset To evaluate on a single dataset, default: 'null'. CODE The code to reproduce the experimental results in the paper is available at https://github.com/LogIntelligence/LogBatcher