Do Code Graphs Help Coding Agents Find Code? A Pre-Registered Evaluation of Type-Resolved Call Graphs for Repository-Level Retrieval
Code-graph tools that plug into AI coding agents through the Model Context Protocol (MCP) report large gains, such as "~63× fewer tokens" or "80% lower agent token spend". The baselines behind those figures are often not the ones a developer has in mind. We built TypeTrace, a local MCP server that combines lexical and dense retrieval with propagation over a call graph resolved by the TypeScript type checker. We evaluated it against graphify and code-review-graph, two popular open-source code-graph tools, in a series of experiments. Every test was pre-registered, and the ground truth does not depend on any of the tools. On three TypeScript repositories from 6k to 763k lines (64 audited tasks), TypeTrace's Recall@10 was 0.33 to 0.69. Neither competitor exceeded 0.31, and 16 of 18 paired comparisons had p ≤ 0.001. On SWE-bench Lite (300 Python issues), Recall@10 was 0.469, against 0.354 for code-review-graph with embeddings and 0.151 for graphify (Holm p ≤ 0.0005). On SWE-bench Multilingual (299 issues in nine other languages), it was 0.389, against 0.261 and 0.123 (Holm p = 0.0001), and TypeTrace led in every language. The call graph's own contribution is conditional. On TypeScript, graph propagation added +0.24, +0.05 and +0.16 Recall@10, concentrated on cross-module tasks (Holm p = 0.019). On SWE-bench it added nothing: −0.018 on Python and −0.002 across the nine other languages, neither significant. We also ran Claude Code agents in 918 headless sessions on held-out tasks. With a ≤800-token, graph-annotated pointer list, the agents used 18–28% fewer tokens than with grep alone on a typical task, in all five runs. Agents given graphify used about as many tokens as grep alone, and agents given code-review-graph used 52–70% more on average. Accuracy gains over grep alone did not replicate across runs. Graph content added no significant accuracy over the same tool without the graph (Holm p = 0.44). The competitors' headline numbers measure different things: the whole repository pasted into the prompt, a hosted product, or conversational memory. We release the tool, the harness, the frozen datasets and the decision logs. This is a preprint of work in progress: two further pre-registered experiments (coding agents on SWE-bench issues; each tool with its vendor's own setup, on each tool's own kind of task) are running and will be added in a later version.
Authors
- Aditya Reddy Singireddy
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24363177
- Primary Topic
- Software Engineering Research
- Type
- preprint