Scaffolded Answers and Process Self-Reports: Two Small Evaluations of an Operational LLM Deployment

External context can specify how a language model should respond, but a response that satisfies operational criteria need not accurately describe the process that produced it. We examine two small evaluations of one operational deployment. In EXP-001, scaffolded outputs scored lower on the accuracy of process self-reports than clean outputs under two scoring procedures (74.6 vs 90.4; 82.7 vs 96.0). In EXP-002, guided answers satisfied more predefined rule criteria than bare answers (mean 0.125 vs 0.875 on eight tasks). The studies use different tasks and endpoints and are not directly comparable; we present them as a bounded case study motivating separate evaluation of answer quality and process-report accuracy. Headline scores were recomputed from the stored scoring files; a full reproducibility package is in preparation. This preprint narrows EFS v3.4 to its two completed evaluations.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-10-15
DOI
https://doi.org/10.5281/zenodo.23390357
Primary Topic
Topic Modeling
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

Scaffolded Answers and Process Self-Reports: Two Small Evaluations of an Operational LLM Deployment

Ondřej Šrámek
Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
preprint

Scaffolded Answers and Process Self-Reports: Two Small Evaluations of an Operational LLM Deployment

Ondřej Šrámek
preprint en

Abstract

External context can specify how a language model should respond, but a response that satisfies operational criteria need not accurately describe the process that produced it. We examine two small evaluations of one operational deployment. In EXP-001, scaffolded outputs scored lower on the accuracy of process self-reports than clean outputs under two scoring procedures (74.6 vs 90.4; 82.7 vs 96.0). In EXP-002, guided answers satisfied more predefined rule criteria than bare answers (mean 0.125 vs 0.875 on eight tasks). The studies use different tasks and endpoints and are not directly comparable; we present them as a bounded case study motivating separate evaluation of answer quality and process-report accuracy. Headline scores were recomputed from the stored scoring files; a full reproducibility package is in preparation. This preprint narrows EFS v3.4 to its two completed evaluations.

Zenodo (CERN European Organization for Nuclear Research)
Topic Modeling
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.