Found but not picked: How code-graph context loses to BM25 in small Gemma 4 windows

Context: Bellam (2026) reports that code-graph tools add 17 to 21 percentage points of strict function Acc@10 over grep alone for 1.7B to 4B models, at a 32k-token budget. Objective: Test whether code-graph context still helps small Gemma 4 models localize issues in small windows, through three research questions: whether it beats BM25 code, whether model size or repo-to-window ratio changes its gain, and why it loses. Method: On 766 issues from 25 Python repositories, fixed after Gemma 4's cutoff, we give four Gemma 4 sizes a flat file tree, BM25 code, or one of two graph layouts at 2k and 8k tokens. Results: BM25 scores above both graph layouts in all 8 model × budget cells, narrowly for 26B at 2k (41.3% vs 40.7%). The graph names the faulty function at least as often as BM25 (coverage 53.7% vs 47.5% at 2k), but Gemma 4 picks it less often (E2B pick rate: 48.4% vs 72.5%): found, but not picked. Running the graph from BM25's seeds closes 3.7 percentage points of BM25's Acc@1 lead on E2B at 2k and 2.3 on E4B at 8k. Each hop from seed to faulty function multiplies the odds by 0.55 to 0.65, and graph layouts deepen that penalty at every size. Conclusion: In small windows the graph finds the faulty function, but Gemma 4 picks it less often. Code and data (Apache-2.0) are in the attached zip. Preprint of the manuscript submitted to Information and Software Technology (Elsevier). An earlier, non-archival version was entered in the Kaggle Gemma 4 Developer Agent Paper Track (2026).

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-13
DOI
https://doi.org/10.5281/zenodo.24144366
Citations
1
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Found but not picked: How code-graph context loses to BM25 in small Gemma 4 windows

Tharun Reddy Kandukuri
1 citations
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Found but not picked: How code-graph context loses to BM25 in small Gemma 4 windows

Tharun Reddy Kandukuri
preprint en
1 citations

Abstract

Context: Bellam (2026) reports that code-graph tools add 17 to 21 percentage points of strict function Acc@10 over grep alone for 1.7B to 4B models, at a 32k-token budget. Objective: Test whether code-graph context still helps small Gemma 4 models localize issues in small windows, through three research questions: whether it beats BM25 code, whether model size or repo-to-window ratio changes its gain, and why it loses. Method: On 766 issues from 25 Python repositories, fixed after Gemma 4's cutoff, we give four Gemma 4 sizes a flat file tree, BM25 code, or one of two graph layouts at 2k and 8k tokens. Results: BM25 scores above both graph layouts in all 8 model × budget cells, narrowly for 26B at 2k (41.3% vs 40.7%). The graph names the faulty function at least as often as BM25 (coverage 53.7% vs 47.5% at 2k), but Gemma 4 picks it less often (E2B pick rate: 48.4% vs 72.5%): found, but not picked. Running the graph from BM25's seeds closes 3.7 percentage points of BM25's Acc@1 lead on E2B at 2k and 2.3 on E4B at 8k. Each hop from seed to faulty function multiplies the odds by 0.55 to 0.65, and graph layouts deepen that penalty at every size. Conclusion: In small windows the graph finds the faulty function, but Gemma 4 picks it less often. Code and data (Apache-2.0) are in the attached zip. Preprint of the manuscript submitted to Information and Software Technology (Elsevier). An earlier, non-archival version was entered in the Kaggle Gemma 4 Developer Agent Paper Track (2026).

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon