Healthy Spans, Failed Agents: Why Trace Sampling Loses the Evidence for LLM Agent Failures

Trace sampling for distributed systems rests on an assumption: the span that explains a failure looks unusual on its own, so rules that keep errors, slow or costly spans, or structurally odd traces also keep the evidence. We show that this assumption does not hold for LLM agents, measure how much diagnostic evidence is lost as a result, and measure what it costs to keep it. On three public agent datasets, one recorded with real OpenTelemetry instrumentation, the span that annotators identify as decisive carries an ERROR status in only 9.8% of Who&When failures and 8.2% of TRAIL failures, and it usually sits several steps before any visible sign of trouble. At equal storage budgets, error, latency and cost rules keep failed runs at about the rate random sampling does, or worse, and anomaly samplers from the trace-sampling literature help only a little. Signals about the run’s outcome do better when they are informative: a metadata classifier keeps 2.6 times as many τ -bench failures as random sampling at a 15% budget, while an LLM judge helps on τ -bench but is no better than chance on TRAIL, where simply keeping the smallest traces retains more failures than any outcome signal we tested. The lost evidence matters: when a sampled view drops the decisive span, an LLM investigator finds the failing step far less often than from the full trace. Keeping that evidence has a systems cost. On the stock OpenTelemetry Collector, outcome-aware retention needs only a configuration change, but agent runs last minutes, so the decision window must too: with a 300 s window, 19% of flagged TRAIL runs are still lost, and collector memory grows with arrival rate, window and trace size. We release the converters, samplers, collector harness and configuration, all LLM outputs, and a onecommand reproduction script.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24298758
Primary Topic
Software System Performance and Reliability
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Healthy Spans, Failed Agents: Why Trace Sampling Loses the Evidence for LLM Agent Failures

Kamal Singh Bisht
Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
preprint

Healthy Spans, Failed Agents: Why Trace Sampling Loses the Evidence for LLM Agent Failures

Kamal Singh Bisht
preprint en

Abstract

Trace sampling for distributed systems rests on an assumption: the span that explains a failure looks unusual on its own, so rules that keep errors, slow or costly spans, or structurally odd traces also keep the evidence. We show that this assumption does not hold for LLM agents, measure how much diagnostic evidence is lost as a result, and measure what it costs to keep it. On three public agent datasets, one recorded with real OpenTelemetry instrumentation, the span that annotators identify as decisive carries an ERROR status in only 9.8% of Who&When failures and 8.2% of TRAIL failures, and it usually sits several steps before any visible sign of trouble. At equal storage budgets, error, latency and cost rules keep failed runs at about the rate random sampling does, or worse, and anomaly samplers from the trace-sampling literature help only a little. Signals about the run’s outcome do better when they are informative: a metadata classifier keeps 2.6 times as many τ -bench failures as random sampling at a 15% budget, while an LLM judge helps on τ -bench but is no better than chance on TRAIL, where simply keeping the smallest traces retains more failures than any outcome signal we tested. The lost evidence matters: when a sampled view drops the decisive span, an LLM investigator finds the failing step far less often than from the full trace. Keeping that evidence has a systems cost. On the stock OpenTelemetry Collector, outcome-aware retention needs only a configuration change, but agent runs last minutes, so the decision window must too: with a 300 s window, 19% of flagged TRAIL runs are still lost, and collector memory grows with arrival rate, window and trace size. We release the converters, samplers, collector harness and configuration, all LLM outputs, and a onecommand reproduction script.

Zenodo (CERN European Organization for Nuclear Research)
Software System Performance and Reliability
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon