No todas las instrucciones se olvidan igual

For an autonomous LLM agent, a safety guardrail is just another instruction given at the start of a run, and as context grows, the agent keeps some instructions while dropping others. Because a mean compliance rate aggregates across instructions, it cannot reveal which one was dropped. We measure retention one instruction at a time by conducting 25-turn coding sessions on five instruction-following models and three open-source Python codebases. In these sessions, we plant twelve ordinary coding preferences as casual inline requests, score the resulting code with deterministic checkers on a 0–3 ordinal scale, and fit a Bayesian ordered-logistic model with hierarchical effects per decision type to 244 observations. Treatment effects span −2.4 to +5.4 on the log-odds scale (group-level standard deviation 2.1), showing considerable variation across preferences. Most preferences are already followed by default, and only three of twelve benefit from being restated (two under a single-model robustness check). Since these preferences stand in for guardrails, this unevenness is a safety concern. Because a guardrail must hold on every turn, aggregate compliance can remain high even after the one rule that matters is skipped. As a result, a deployed agent needs per-instruction monitoring to catch it.

Authors

Institutions

Publication Details

Journal
JAIIO, Jornadas Argentinas de Informática
Published
2026-10-13
Primary Topic
Ethics and Social Impacts of AI
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

No todas las instrucciones se olvidan igual

Tomás Pablo Korenblit
JAIIO, Jornadas Argentinas de Informática
Ethics and Social Impacts of AI
article

No todas las instrucciones se olvidan igual

Tomás Pablo Korenblit
article en

Abstract

For an autonomous LLM agent, a safety guardrail is just another instruction given at the start of a run, and as context grows, the agent keeps some instructions while dropping others. Because a mean compliance rate aggregates across instructions, it cannot reveal which one was dropped. We measure retention one instruction at a time by conducting 25-turn coding sessions on five instruction-following models and three open-source Python codebases. In these sessions, we plant twelve ordinary coding preferences as casual inline requests, score the resulting code with deterministic checkers on a 0–3 ordinal scale, and fit a Bayesian ordered-logistic model with hierarchical effects per decision type to 244 observations. Treatment effects span −2.4 to +5.4 on the log-odds scale (group-level standard deviation 2.1), showing considerable variation across preferences. Most preferences are already followed by default, and only three of twelve benefit from being restated (two under a single-model robustness check). Since these preferences stand in for guardrails, this unevenness is a safety concern. Because a guardrail must hold on every turn, aggregate compliance can remain high even after the one rule that matters is skipped. As a result, a deployed agent needs per-instruction monitoring to catch it.

JAIIO, Jornadas Argentinas de Informática
National University of General San Martín (AR)
Openalex Percentile: Top 7%
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.