Beyond Prompt Engineering: A controlled Evaluation of Deterministic Guardrails in a Transactional LLM Agent

Large language models are increasingly used as conversational agents that interact with databases, APIs, and business systems. Although detail prompting can constrain model behavior, transactional applications require more than natural-language correctness: prices, delivery coverage, routing validation, and external actions must correspond to authoritative system state. This study compares two resturant-ordering agents build using the same language model configuration. System A is an LLM-only baseline supplied with a static snapshot of resturant information through its system prompt, with no tools, databse access, or deterministic business logic. System B uses the same model but augments it with live database-backed tools, deterministic delivery routing, and validation before transactional execution. Both systems were evaluated on 20 controlled scenarios covering menu lookup, incorrect customer assertions, item variations, delivery coverage, payment restrictions, policy compliance, prompt injection, recommendations, and end-to-end order handling. The prompt-only system successfully completed 14 of 20 scenarios (70%), while the tool-grounded system completed 18 of 20 (90%). However, the tool-grounded architecture remained susceptible to intent-routing and cross-turn state-management failures. In the end-to-end transaction, deterministic validation rejected a malformed routing payload and prevented an invalid order from being created. These results suggest that deterministic tools do not eliminate LLM errors. Instead, their main reliability benefit lies in grounding dynamic business deicisions in authoritative state and preventing incorrect model outputs from propagating into consequential external actions.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-09-14
DOI
https://doi.org/10.5281/zenodo.22743668
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
preprint

Beyond Prompt Engineering: A controlled Evaluation of Deterministic Guardrails in a Transactional LLM Agent

Aman Malik
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Beyond Prompt Engineering: A controlled Evaluation of Deterministic Guardrails in a Transactional LLM Agent

Aman Malik
preprint en

Abstract

Large language models are increasingly used as conversational agents that interact with databases, APIs, and business systems. Although detail prompting can constrain model behavior, transactional applications require more than natural-language correctness: prices, delivery coverage, routing validation, and external actions must correspond to authoritative system state. This study compares two resturant-ordering agents build using the same language model configuration. System A is an LLM-only baseline supplied with a static snapshot of resturant information through its system prompt, with no tools, databse access, or deterministic business logic. System B uses the same model but augments it with live database-backed tools, deterministic delivery routing, and validation before transactional execution. Both systems were evaluated on 20 controlled scenarios covering menu lookup, incorrect customer assertions, item variations, delivery coverage, payment restrictions, policy compliance, prompt injection, recommendations, and end-to-end order handling. The prompt-only system successfully completed 14 of 20 scenarios (70%), while the tool-grounded system completed 18 of 20 (90%). However, the tool-grounded architecture remained susceptible to intent-routing and cross-turn state-management failures. In the end-to-end transaction, deterministic validation rejected a malformed routing payload and prevented an invalid order from being created. These results suggest that deterministic tools do not eliminate LLM errors. Instead, their main reliability benefit lies in grounding dynamic business deicisions in authoritative state and preventing incorrect model outputs from propagating into consequential external actions.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.