From BM25 to LLMs: Benchmarking Strategies for Bug Localization at Repository Scale

Embeddings and LLM-Based Bug localization The task is: given an issue with the bug description and the repository code in the state where the issue is reproducible, identify the files within the project that need to be modified to address the reported bug. 💾 Install dependencies We provide dependencies for the pip dependency manager, so please run the following command to install all the required packages:pip install -r requirements.txt 🤗 Datasets Data for both datasets is available in HuggingFace: Long Code Arena (LCA): https://huggingface.co/datasets/JetBrains-Research/lca-bug-localizationSWE-bench Verified (SWE): https://huggingface.co/datasets/princeton-nlp/SWE-bench_Verified Techniques This section details how to run each technique. RQ1 Retrieval BM25: Follow instructions on how to install Pyserini: https://github.com/castorini/pyserini. Configure your running environment in bug_localization/configs/baselines/bm25.yaml. All scripts for BM25 are located under bug_localization. Run with src/baselines/run_bm25.py (for LCA) or src/baselines/run_bm25_swe.py (for SWE-bench Verified). Calculate metrics: src/baselines/metrics/compute_metrics_bm25.py GTE-Large: All scripts for GTE-Large experiments are located under gte-large. The databases with all embeddings for LCA and SWE are located under databases/embeddings. HNSW/chroma_db-based: use rag_pipeline.py LSH/FLAT-L2: use rag_pipeline_faiss.py Compute metrics: compute_metrics.py RQ2 LLM-Based experiments All configuration files are located under bug_localization/config/baselines. Please make sure to set the configuration for the experiment you want to run in the entry run_* file. Run files are located under bug_localization/src/baselines. Bounded-k experiment (BKP): Configure your runs in config/baselines (per-model). Run with run_baseline(_swe).py. Unbounded k experiments (UBP): Configure qwen-agent[xx].yaml (LCA) or swe-agent[xx].yaml (SWE), where you replace [xx] by the name of the strategy (ubp, ubp_rank_summaries, ubp_file_summaries). Run with run_agents(_swe).py. File summaries generation (for rank with summaries and file summaries experiments): Run with summary-generation/rag_pipeline.py Discussion point: Hybrid approaches BM25+Rank with summaries All files are located under bug_localization. Configure ranker.yaml. Run with run_summaries_ranker(_swe).py. BM25+LLM Judge All files are located under bug_localization. Use run_llm_verifier.py (arguments required are documented in the file). BM25 with dependencies Run run_bm25_dep_compare(_swe).py GTE-Large with dependencies See gte-large/rag_pipeline[_swe].py for command-line options. Results All results can be found under the results directory, with a subdirectory for each RQ.