Evaluating Code Navigation Tools for AI Agents against the Compilers and Test Suites of Open-Source Projects
AI coding agents rely on code navigation tools: who calls a function, which tests cover it, what a rename affects. The accuracy of such answers is rarely measured, since common benchmarks score the agent as a whole. We propose a method in which the ground truth comes from the project’s own compiler and tests. Callers are derived from the compile errors that appear after the definition is renamed, a rename is correct if the project still builds and passes its tests, and the tests of a function are those that fail when the function is broken. Every number carries a noise threshold (bootstrap), and two versions of a tool are compared on the same functions. The method is applied to the open- source tool gitnext on ten projects in five languages (232 functions for callers, 338 renames, 167 functions for tests). Callers are found in 100% ±1 of cases with a precision of 99% ±1; after a rename the project builds and passes its tests in 91% ±3 of cases; at least one of the first ten tests named fails in 79% ±6 of cases. The method exposed defects invisible on a small sample and errors in the ground truth itself. In addition, a per-test coverage map raises the share of seeded bugs where the function at fault is named first from 18% to 81% (57 bugs), and a 4-billion-parameter local model solves 17 of 24 tasks with the tool against 4 without it.
Authors
- Leonid Kononenko (ORCID: https://orcid.org/0009-0009-8407-9214)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.5281/zenodo.24172348
- Primary Topic
- Software Engineering Research
- Type
- preprint