Measuring a Scheduling Agent Honestly: Failure Modes of Model-Driven Agent Loops and a Blind-Set Evaluation of a Read-Only-Model Design

Agent frameworks commonly place a large language model in the control loop: the model chooses tools, sequences them, and decides when a task is complete. We report on operating such a design for meeting scheduling, a domain where most errors are visible to people outside the organization and cannot be undone. From our engineering record we derive a taxonomy of eight recurring failure modes, and observe that every mitigation we shipped took a decision away from the model and added latency and cost. We then describe a replacement design in which the model's only role is to read a request into a fixed, validated form, and all planning, time arithmetic and execution are ordinary code behind a single human approval gate. Finally we describe an evaluation protocol built to resist self-deception (situations rather than sentences, independently written phrasings, a held-back half and a blind final set, three runs per phrasing, and release-blocking serious errors) and report current results without adjustment: 93.8% on everyday requests and 94.0% on requests involving outside guests on the blind final set, against our own launch target of 99.5%, with zero serious errors in either suite.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-17
DOI
https://doi.org/10.5281/zenodo.24324273
Primary Topic
Artificial Intelligence Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Measuring a Scheduling Agent Honestly: Failure Modes of Model-Driven Agent Loops and a Blind-Set Evaluation of a Read-Only-Model Design

Raj Lal
5 links
Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence Applications
preprint

Measuring a Scheduling Agent Honestly: Failure Modes of Model-Driven Agent Loops and a Blind-Set Evaluation of a Read-Only-Model Design

Raj Lal
preprint en

Abstract

Agent frameworks commonly place a large language model in the control loop: the model chooses tools, sequences them, and decides when a task is complete. We report on operating such a design for meeting scheduling, a domain where most errors are visible to people outside the organization and cannot be undone. From our engineering record we derive a taxonomy of eight recurring failure modes, and observe that every mitigation we shipped took a decision away from the model and added latency and cost. We then describe a replacement design in which the model's only role is to read a request into a fixed, validated form, and all planning, time arithmetic and execution are ordinary code behind a single human approval gate. Finally we describe an evaluation protocol built to resist self-deception (situations rather than sentences, independently written phrasings, a held-back half and a blind final set, three runs per phrasing, and release-blocking serious errors) and report current results without adjustment: 93.8% on everyday requests and 94.0% on requests involving outside guests on the blind final set, against our own launch target of 99.5%, with zero serious errors in either suite.

Zenodo (CERN European Organization for Nuclear Research)
Artificial Intelligence Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon