Experimental Multi-Metric Evaluation of Interactive LLM Agents - Efficiency, Repeated-Trajectory Variability, and Ranking Sensitivity
—Official task success alone leaves an agent’s resource use, execution behavior, and variation across repeated runs unreported. This thesis evaluates whether controlled configuration changes to an interactive large language model agent alter official task success, trajectory efficiency, execution validity, and variation across repeated runs. A frozen GPT-5 snapshot was tested in DBBench and OSBench using a 2 ×2 factorial design that varied the budget for interaction rounds and the output constraint. Five repetitions of every task–condition combination produced 8,880 analyzed episodes. Task-clustered models identified no clear population-average contrast in official success for either factor or their interaction in either environment. Secondary results differed: the C1 condition, which combines a strict parser with one repair opportunity, was associated with increased DBBench steps and token use, whereas OSBench showed adjusted associations for framework completion but not official correctness. DBBench final answers were more repeatable than exact action sequences, and trajectory features improved grouped prediction without supporting a causal interpretation. Configuration rankings across the two environments changed when trajectory measures were added to success, but the leading configuration depended on the aggregation scheme. The results support reporting endpoint, trajectory, execution, and variability measures separately rather than treating one composite ordering as definitive for the evaluated settings.
Authors
- Fatemeh Akbarifar
Publication Details
- Journal
- Gothenburg University Publications Electronic Archive (Gothenburg University)
- Published
- 2026-11-13
- Primary Topic
- AI-based Problem Solving and Planning
- Type
- article
- Field-Weighted Citation Impact
- 0.00