Reliability Engineering for Pass³ Tool Use on CAR-bench Track 1: Faithful Simulation, Live-Schema Guarding, and Gated Admission
We report the Track 1 system of team Thylinao on CAR-bench, whose Pass³ metric rewards an agent only when all three trials of a task pass every applicable sub-metric. Starting from the provided LiteLLM tool-calling agent, three validated changes (greedy decoding, a compliant live-schema guard, and an agent-model upgrade to gemini-3.5-flash, the dominant lever, with the guard as low-cost insurance) plus one general behaviour skill at high reasoning effort reach 80.2% Pass³ on the complete 129-task training split, +18.9 points above the strongest published reference on the identical tasks, and 78.0% in a single pre-registered read of the held-out public test split, where the references we recompute top out at 58%. Both margins are cross-era, and our calibration probes indicate the era gap favours the references. The headline is performance per unit of model: a flash-class agent clears every frontier reference by more than 18 points; the lead costs latency and dollars, and we report the full trade. Beyond the score, two findings transfer to any simulator-judge Passᵏ benchmark: a provider-route pitfall that swings hallucination by about 25 points if uncontrolled, and a gated-admission discipline whose full-split re-measurement caught a prompt-interference tax, where added prompt text degrades rules it never touches, invisible to targeted probes. Technical report of the 2nd-placed entry (of 21 teams) in Track 1 (Open) of the CAR-bench Challenge at IJCAI-ECAI 2026, team Thylinao. The body text is identical to the report published by the organizers at https://car-bench.github.io/car-bench/reports/track_1/team-28.pdf.
Authors
- Maksim Silchenko
Institutions
- National University of Singapore (SG)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-08-25
- DOI
- https://doi.org/10.5281/zenodo.22092522
- Primary Topic
- Human-Automation Interaction and Safety
- Type
- article
- Field-Weighted Citation Impact
- 0.00