Injected Recommendations Are Not Enforcement: A Blind Census of Pointer Relevance and Pointer Use in a Failure-Derived Constraint System

Rules that fire on observed failures are a common dependability mechanism for LLM agents: a trigger executes outside the model, so a catalogued error cannot recur without something in the pipeline firing first. We report five months of measurement on such a system running inside a working engineering practice, with 21 active rules, 9,022 recorded constraint events and a memory corpus of 949 files. The highest-volume rule is a memory recall hook. On each user prompt it matches the prompt against an index of one-line pointers and injects the names of the closest memory files into the turn; it never injects their content. We took every hash-verified firing of that hook in two windows, 535 before and 699 after a filter that removed automatic system events from the trigger, and judged all 1,234 rows on one question, blind to window and outcome: was any recommended pointer relevant to what the prompt needed? Judgments came from a single model, with a fixed-seed blind re-labeling of 60 rows giving intra-rater agreement of 93.3 % (Cohen's kappa 0.837). A deterministic pass over session transcripts then recorded whether any recommended file was subsequently read. On human-authored prompts the pointer was relevant in 683 of 1,000 firings (68.3 %). In the post-filter window, where the mechanical signal is complete, a relevant pointer was followed by a read of some recommended file in 123 of 471 cases (26.1 %) and by a read of the relevant file itself in 111 (23.6 %). When the pointer was not relevant, a recommended file was still read in 10.6 % (pre) and 11.0 % (post) of human-prompt cases; reading was associated with pointer relevance, and the corrective action itself was not measured. The filter itself moved firings on automatic events from 234 of 535 to 0 of 699, counted as a form rather than a judgment. The result separates two events usually treated as one. A trigger that executes independently of model discipline delivers a pointer independently of model discipline; whether the pointed-to content enters the context, and whether the corrective action follows, remain decisions of the model. We state the boundary, add measured precision with a demotion threshold to the requirements for such a mechanism, and report the practices that made the measurement possible at all.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-11
DOI
https://doi.org/10.5281/zenodo.22709602
Primary Topic
Multi-Agent Systems and Negotiation
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Injected Recommendations Are Not Enforcement: A Blind Census of Pointer Relevance and Pointer Use in a Failure-Derived Constraint System

Victor Del Puerto
Zenodo (CERN European Organization for Nuclear Research)
Multi-Agent Systems and Negotiation
preprint

Injected Recommendations Are Not Enforcement: A Blind Census of Pointer Relevance and Pointer Use in a Failure-Derived Constraint System

Victor Del Puerto
preprint en

Abstract

Rules that fire on observed failures are a common dependability mechanism for LLM agents: a trigger executes outside the model, so a catalogued error cannot recur without something in the pipeline firing first. We report five months of measurement on such a system running inside a working engineering practice, with 21 active rules, 9,022 recorded constraint events and a memory corpus of 949 files. The highest-volume rule is a memory recall hook. On each user prompt it matches the prompt against an index of one-line pointers and injects the names of the closest memory files into the turn; it never injects their content. We took every hash-verified firing of that hook in two windows, 535 before and 699 after a filter that removed automatic system events from the trigger, and judged all 1,234 rows on one question, blind to window and outcome: was any recommended pointer relevant to what the prompt needed? Judgments came from a single model, with a fixed-seed blind re-labeling of 60 rows giving intra-rater agreement of 93.3 % (Cohen's kappa 0.837). A deterministic pass over session transcripts then recorded whether any recommended file was subsequently read. On human-authored prompts the pointer was relevant in 683 of 1,000 firings (68.3 %). In the post-filter window, where the mechanical signal is complete, a relevant pointer was followed by a read of some recommended file in 123 of 471 cases (26.1 %) and by a read of the relevant file itself in 111 (23.6 %). When the pointer was not relevant, a recommended file was still read in 10.6 % (pre) and 11.0 % (post) of human-prompt cases; reading was associated with pointer relevance, and the corrective action itself was not measured. The filter itself moved firings on automatic events from 234 of 535 to 0 of 699, counted as a form rather than a judgment. The result separates two events usually treated as one. A trigger that executes independently of model discipline delivers a pointer independently of model discipline; whether the pointed-to content enters the context, and whether the corrective action follows, remain decisions of the model. We state the boundary, add measured precision with a demotion threshold to the requirements for such a mechanism, and report the practices that made the measurement possible at all.

Zenodo (CERN European Organization for Nuclear Research)
Técnicas y Servicios de Ingeniería (Spain) (ES)
Peace, Justice and strong institutions
Multi-Agent Systems and Negotiation
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.