Experimental Multi-Metric Evaluation of Interactive LLM Agents - Efficiency, Repeated-Trajectory Variability, and Ranking Sensitivity

—Official task success alone leaves an agent’s resource use, execution behavior, and variation across repeated runs unreported. This thesis evaluates whether controlled configuration changes to an interactive large language model agent alter official task success, trajectory efficiency, execution validity, and variation across repeated runs. A frozen GPT-5 snapshot was tested in DBBench and OSBench using a 2 ×2 factorial design that varied the budget for interaction rounds and the output constraint. Five repetitions of every task–condition combination produced 8,880 analyzed episodes. Task-clustered models identified no clear population-average contrast in official success for either factor or their interaction in either environment. Secondary results differed: the C1 condition, which combines a strict parser with one repair opportunity, was associated with increased DBBench steps and token use, whereas OSBench showed adjusted associations for framework completion but not official correctness. DBBench final answers were more repeatable than exact action sequences, and trajectory features improved grouped prediction without supporting a causal interpretation. Configuration rankings across the two environments changed when trajectory measures were added to success, but the leading configuration depended on the aggregation scheme. The results support reporting endpoint, trajectory, execution, and variability measures separately rather than treating one composite ordering as definitive for the evaluated settings.

Authors

Publication Details

Journal
Gothenburg University Publications Electronic Archive (Gothenburg University)
Published
2026-11-13
Primary Topic
AI-based Problem Solving and Planning
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Experimental Multi-Metric Evaluation of Interactive LLM Agents - Efficiency, Repeated-Trajectory Variability, and Ranking Sensitivity

Fatemeh Akbarifar
Gothenburg University Publications Electronic Archive (Gothenburg University)
AI-based Problem Solving and Planning
article

Experimental Multi-Metric Evaluation of Interactive LLM Agents - Efficiency, Repeated-Trajectory Variability, and Ranking Sensitivity

Fatemeh Akbarifar
article en

Abstract

—Official task success alone leaves an agent’s resource use, execution behavior, and variation across repeated runs unreported. This thesis evaluates whether controlled configuration changes to an interactive large language model agent alter official task success, trajectory efficiency, execution validity, and variation across repeated runs. A frozen GPT-5 snapshot was tested in DBBench and OSBench using a 2 ×2 factorial design that varied the budget for interaction rounds and the output constraint. Five repetitions of every task–condition combination produced 8,880 analyzed episodes. Task-clustered models identified no clear population-average contrast in official success for either factor or their interaction in either environment. Secondary results differed: the C1 condition, which combines a strict parser with one repair opportunity, was associated with increased DBBench steps and token use, whereas OSBench showed adjusted associations for framework completion but not official correctness. DBBench final answers were more repeatable than exact action sequences, and trajectory features improved grouped prediction without supporting a causal interpretation. Configuration rankings across the two environments changed when trajectory measures were added to success, but the leading configuration depended on the aggregation scheme. The results support reporting endpoint, trajectory, execution, and variability measures separately rather than treating one composite ordering as definitive for the evaluated settings.

Gothenburg University Publications Electronic Archive (Gothenburg University)
AI-based Problem Solving and Planning
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon