Human-Rated Quality and Measured Token Use in Organizational RAG: A Field Comparison of Three Configurations
Retrieval-augmented generation (RAG) is commonly assessed on benchmarks, but its behavior under employees’ workplace questions remains less clear. We compared single-pass, agentic, and hybrid RAG in a Slovenian public-sector assistant to examine user-perceived quality, actual tool use, and token consumption. Configurations were randomized per turn, and their names were not shown to users. We analyzed 701 responses from 131 accounts; 402 answers were evaluated by 73 accounts using five 1–5 rating items and a separate binary usefulness judgment. Resource use was application-recorded input plus output tokens, excluding re-ranking. Hybrid had the highest descriptive mean quality (4.15), followed by Single-pass (3.93) and Agentic (3.79), but neither dimension tests nor adjusted composite contrasts established superiority. Hybrid also had the highest mean token total among rated responses (41,143 versus 32,977 and 28,432). Agentic initiated document retrieval on 62.9% of turns; Hybrid used additional retrieval in 3.5%. Thus, the comparison showed a design–behavior gap without establishing an escalation benefit. The study provides field-evaluation evidence and practical deployment recommendations; incomplete rating coverage, configuration-specific prompts, and mixed conversation histories limit generalization and causal interpretation.
Authors
- Robert Leskovar (ORCID: https://orcid.org/0000-0002-5325-7542)
- Matjaž Gams (ORCID: https://orcid.org/0000-0002-5747-0711)
- Mihael Škarabot
Institutions
- Jožef Stefan Institute (SI)
- University of Maribor (SI)
- Jožef Stefan International Postgraduate School (SI)
Publication Details
- Journal
- Applied Sciences
- Published
- 2026-11-19
- DOI
- https://doi.org/10.3390/app162211481
- Primary Topic
- Information Retrieval and Search Behavior
- Type
- article
- Field-Weighted Citation Impact
- 0.00