Evaluating Code Navigation Tools for AI Agents against the Compilers and Test Suites of Open-Source Projects

AI coding agents rely on code navigation tools: who calls a function, which tests cover it, what a rename affects. The accuracy of such answers is rarely measured, since common benchmarks score the agent as a whole. We propose a method in which the ground truth comes from the project’s own compiler and tests. Callers are derived from the compile errors that appear after the definition is renamed, a rename is correct if the project still builds and passes its tests, and the tests of a function are those that fail when the function is broken. Every number carries a noise threshold (bootstrap), and two versions of a tool are compared on the same functions. The method is applied to the open- source tool gitnext on ten projects in five languages (232 functions for callers, 338 renames, 167 functions for tests). Callers are found in 100% ±1 of cases with a precision of 99% ±1; after a rename the project builds and passes its tests in 91% ±3 of cases; at least one of the first ten tests named fails in 79% ±6 of cases. The method exposed defects invisible on a small sample and errors in the ground truth itself. In addition, a per-test coverage map raises the share of seeded bugs where the function at fault is named first from 18% to 81% (57 bugs), and a 4-billion-parameter local model solves 17 of 24 tasks with the tool against 4 without it.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-12
DOI
https://doi.org/10.5281/zenodo.24172348
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Evaluating Code Navigation Tools for AI Agents against the Compilers and Test Suites of Open-Source Projects

Leonid Kononenko
Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
preprint

Evaluating Code Navigation Tools for AI Agents against the Compilers and Test Suites of Open-Source Projects

Leonid Kononenko
preprint en

Abstract

AI coding agents rely on code navigation tools: who calls a function, which tests cover it, what a rename affects. The accuracy of such answers is rarely measured, since common benchmarks score the agent as a whole. We propose a method in which the ground truth comes from the project’s own compiler and tests. Callers are derived from the compile errors that appear after the definition is renamed, a rename is correct if the project still builds and passes its tests, and the tests of a function are those that fail when the function is broken. Every number carries a noise threshold (bootstrap), and two versions of a tool are compared on the same functions. The method is applied to the open- source tool gitnext on ten projects in five languages (232 functions for callers, 338 renames, 167 functions for tests). Callers are found in 100% ±1 of cases with a precision of 99% ±1; after a rename the project builds and passes its tests in 91% ±3 of cases; at least one of the first ten tests named fails in 79% ±6 of cases. The method exposed defects invisible on a small sample and errors in the ground truth itself. In addition, a per-test coverage map raises the share of seeded bugs where the function at fault is named first from 18% to 81% (57 bugs), and a 4-billion-parameter local model solves 17 of 24 tasks with the tool against 4 without it.

Zenodo (CERN European Organization for Nuclear Research)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon