Reliability Engineering for Pass³ Tool Use on CAR-bench Track 1: Faithful Simulation, Live-Schema Guarding, and Gated Admission

We report the Track 1 system of team Thylinao on CAR-bench, whose Pass³ metric rewards an agent only when all three trials of a task pass every applicable sub-metric. Starting from the provided LiteLLM tool-calling agent, three validated changes (greedy decoding, a compliant live-schema guard, and an agent-model upgrade to gemini-3.5-flash, the dominant lever, with the guard as low-cost insurance) plus one general behaviour skill at high reasoning effort reach 80.2% Pass³ on the complete 129-task training split, +18.9 points above the strongest published reference on the identical tasks, and 78.0% in a single pre-registered read of the held-out public test split, where the references we recompute top out at 58%. Both margins are cross-era, and our calibration probes indicate the era gap favours the references. The headline is performance per unit of model: a flash-class agent clears every frontier reference by more than 18 points; the lead costs latency and dollars, and we report the full trade. Beyond the score, two findings transfer to any simulator-judge Passᵏ benchmark: a provider-route pitfall that swings hallucination by about 25 points if uncontrolled, and a gated-admission discipline whose full-split re-measurement caught a prompt-interference tax, where added prompt text degrades rules it never touches, invisible to targeted probes. Technical report of the 2nd-placed entry (of 21 teams) in Track 1 (Open) of the CAR-bench Challenge at IJCAI-ECAI 2026, team Thylinao. The body text is identical to the report published by the organizers at https://car-bench.github.io/car-bench/reports/track_1/team-28.pdf.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-08-25
DOI
https://doi.org/10.5281/zenodo.22092523
Primary Topic
Human-Automation Interaction and Safety
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Reliability Engineering for Pass³ Tool Use on CAR-bench Track 1: Faithful Simulation, Live-Schema Guarding, and Gated Admission

Maksim Silchenko
Zenodo (CERN European Organization for Nuclear Research)
Human-Automation Interaction and Safety
article

Reliability Engineering for Pass³ Tool Use on CAR-bench Track 1: Faithful Simulation, Live-Schema Guarding, and Gated Admission

Maksim Silchenko
article en

Abstract

We report the Track 1 system of team Thylinao on CAR-bench, whose Pass³ metric rewards an agent only when all three trials of a task pass every applicable sub-metric. Starting from the provided LiteLLM tool-calling agent, three validated changes (greedy decoding, a compliant live-schema guard, and an agent-model upgrade to gemini-3.5-flash, the dominant lever, with the guard as low-cost insurance) plus one general behaviour skill at high reasoning effort reach 80.2% Pass³ on the complete 129-task training split, +18.9 points above the strongest published reference on the identical tasks, and 78.0% in a single pre-registered read of the held-out public test split, where the references we recompute top out at 58%. Both margins are cross-era, and our calibration probes indicate the era gap favours the references. The headline is performance per unit of model: a flash-class agent clears every frontier reference by more than 18 points; the lead costs latency and dollars, and we report the full trade. Beyond the score, two findings transfer to any simulator-judge Passᵏ benchmark: a provider-route pitfall that swings hallucination by about 25 points if uncontrolled, and a gated-admission discipline whose full-split re-measurement caught a prompt-interference tax, where added prompt text degrades rules it never touches, invisible to targeted probes. Technical report of the 2nd-placed entry (of 21 teams) in Track 1 (Open) of the CAR-bench Challenge at IJCAI-ECAI 2026, team Thylinao. The body text is identical to the report published by the organizers at https://car-bench.github.io/car-bench/reports/track_1/team-28.pdf.

Zenodo (CERN European Organization for Nuclear Research)
National University of Singapore (SG)
Openalex Percentile: Top 6%
Human-Automation Interaction and Safety
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Reliability Engineering for Pass³ Tool Use on CAR-bench Track 1: Faithful Simulation, Live-Schema Guarding, and Gated Admission — Maksim Silchenko · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS