Beyond Schema Conformance with Executable Verification for Scientific Configuration Repair

Abstract Large language models can generate structurally valid scientific configuration files, but whether those configurations reproduce the intended computation is rarely tested. We study this gap on Palace, a finite-element solver for computa- tional electromagnetics, using an executable verification hierarchy that measures schema conformance, operational acceptance by the solver, and numerical ref- erence conformance. We construct a source-grounded fine-tuning corpus and a configuration-repair benchmark containing both shipped cases represented in the training corpus and newly authored cases created after the model and retrieval index were frozen. Fine-tuning provides the largest improvement, while the RAG-proxy setting and schema-guided repair provide additional gains. However, these im- provements transfer much more strongly at the structural level than at the numerical level. On newly authored cases, the full pipeline reaches 89% schema conformance and 81% solver acceptance, but only 41% reference conformance. We further show that some failures arise from insufficient task specification and that blind LLM judges do not consistently reproduce execution-based labels. These results suggest that scientific configuration generation and repair should be evaluated against executable numerical references rather than schema conformance or model judgment alone.

Authors

Publication Details

Journal
Research Square
Published
2026-11-17
DOI
https://doi.org/10.21203/rs.3.rs-11746019/v1
Primary Topic
Software Engineering Research
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Beyond Schema Conformance with Executable Verification for Scientific Configuration Repair

Shoaib Noor, Wasif Masood
2 links
Research Square
Software Engineering Research
preprint

Beyond Schema Conformance with Executable Verification for Scientific Configuration Repair

Shoaib Noor, Wasif Masood
preprint en

Abstract

Abstract Large language models can generate structurally valid scientific configuration files, but whether those configurations reproduce the intended computation is rarely tested. We study this gap on Palace, a finite-element solver for computa- tional electromagnetics, using an executable verification hierarchy that measures schema conformance, operational acceptance by the solver, and numerical ref- erence conformance. We construct a source-grounded fine-tuning corpus and a configuration-repair benchmark containing both shipped cases represented in the training corpus and newly authored cases created after the model and retrieval index were frozen. Fine-tuning provides the largest improvement, while the RAG-proxy setting and schema-guided repair provide additional gains. However, these im- provements transfer much more strongly at the structural level than at the numerical level. On newly authored cases, the full pipeline reaches 89% schema conformance and 81% solver acceptance, but only 41% reference conformance. We further show that some failures arise from insufficient task specification and that blind LLM judges do not consistently reproduce execution-based labels. These results suggest that scientific configuration generation and repair should be evaluated against executable numerical references rather than schema conformance or model judgment alone.

Research Square
Österreichische Forschungsförderungsgesellschaft
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon