Who Speaks for Whom: Prompt Injection and Authority in an Email-Drafting Agent

AI systems now act for people: they draft, reply, book and pay. That makes authority a practical question, not an abstract one: who may direct the system, over what, shown how, and for how long. We give a four-part working definition (source, scope, proof, revocation) and illustrate how each part fails. We then test one failure in our own product. Orchestrate drafts email replies for small businesses, and an owner approves each draft before it is sent. We hid 12 manipulative instructions in prospect replies and ran each 3 times on three models, including the production model. Two graders from different providers agreed on 91% of drafts (Cohen's κ = 0.87). By their joint judgement, the production model obeyed the attacker in 14 of 24 drafts (58%; 95% CI 39–76%); the provider's content filter blocked the other 12 requests. A hand check by the author on 60 drafts agreed with the graders on 93% of clearly harmful attacks (κ = 0.78) but only 63% of ambiguous ones. We deployed two changes: the prospect's words were fenced off as untrusted, and a check outside the model held any draft stating facts absent from the business's records. A third change held drafts that claim actions never taken. After the fix the production model still followed the attacker in 11 of 30 and then 6 of 30 drafts across two runs (37% and 20%), Gemini in 1 of 36 and Claude in none. The checks held every draft that both graders judged compliant before it reached the owner as a ready answer. Three drafts flagged by only one grader were not held; all three asked before acting. Fencing cut compliance sharply for Gemini and Claude but not clearly for the production model. The checks outside the model, not fencing, are what kept compliant drafts from reaching owners as ready answers. YaktAI Lab Report 1. Preprint, not peer-reviewed. A follow-up, Report 2, tests the same check against reworded attacks.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24271286
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Who Speaks for Whom: Prompt Injection and Authority in an Email-Drafting Agent

Muhammad Sakhawat
6 links
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

Who Speaks for Whom: Prompt Injection and Authority in an Email-Drafting Agent

Muhammad Sakhawat
preprint en

Abstract

AI systems now act for people: they draft, reply, book and pay. That makes authority a practical question, not an abstract one: who may direct the system, over what, shown how, and for how long. We give a four-part working definition (source, scope, proof, revocation) and illustrate how each part fails. We then test one failure in our own product. Orchestrate drafts email replies for small businesses, and an owner approves each draft before it is sent. We hid 12 manipulative instructions in prospect replies and ran each 3 times on three models, including the production model. Two graders from different providers agreed on 91% of drafts (Cohen's κ = 0.87). By their joint judgement, the production model obeyed the attacker in 14 of 24 drafts (58%; 95% CI 39–76%); the provider's content filter blocked the other 12 requests. A hand check by the author on 60 drafts agreed with the graders on 93% of clearly harmful attacks (κ = 0.78) but only 63% of ambiguous ones. We deployed two changes: the prospect's words were fenced off as untrusted, and a check outside the model held any draft stating facts absent from the business's records. A third change held drafts that claim actions never taken. After the fix the production model still followed the attacker in 11 of 30 and then 6 of 30 drafts across two runs (37% and 20%), Gemini in 1 of 36 and Claude in none. The checks held every draft that both graders judged compliant before it reached the owner as a ready answer. Three drafts flagged by only one grader were not held; all three asked before acting. Fencing cut compliance sharply for Gemini and Claude but not clearly for the production model. The checks outside the model, not fencing, are what kept compliant drafts from reaching owners as ready answers. YaktAI Lab Report 1. Preprint, not peer-reviewed. A follow-up, Report 2, tests the same check against reworded attacks.

Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon