Executable Verification Framework for Benchmarking AI-Based Decision Support in Physically Network-Constrained Energy Systems

Benchmarks of inverter-based power grids that use machine learning (ML), large language models (LLMs), and learning-based inverter control can be misleading when fluent responses are operationally infeasible, judge-based scores are biased, or forecast-accuracy metrics such as Mean Absolute Error (MAE) do not reflect reserve and dispatch consequences. This study introduces VERITAS-GridBench, an operationally grounded benchmark in which decision-support outputs are evaluated against simulator/optimizer-based "Executable Truth" using explicit feasibility and constraint-violation measures. The framework combines tool verification, rule-based checks, and calibrated multi-judge semantic assessment through Triangulated Scoring, while also incorporating reproducibility, contamination control, domain-shift testing, and a Forecast-to-Operations (F2O) pathway. The experiments compare judge-only, multi-judge, and triangulated evaluation for power-system decision support and compare representative statistical, machine-learning, sequence, convolutional, and transformer forecasting models after their forecasts are propagated to reserve and dispatch calculations. The results show that judge-only scoring can reward fluent but infeasible actions, whereas tool-verified triangulation markedly increases feasibility and reduces unsafe wins and constraint violations. The forecasting results further show that lower MAE does not necessarily imply lower operational cost or risk. These findings support end-to-end evaluation in which model quality is judged by both predictive performance and physically executable grid outcomes.

Authors

Institutions

Publication Details

Journal
Recent Innovations in Mechatronics
Published
2026-11-18
DOI
https://doi.org/10.17667/riim.2026.1
Primary Topic
Smart Grid and Power Systems
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Executable Verification Framework for Benchmarking AI-Based Decision Support in Physically Network-Constrained Energy Systems

Shafaa mahmood shnawa
2 links
Recent Innovations in Mechatronics
Smart Grid and Power Systems
article

Executable Verification Framework for Benchmarking AI-Based Decision Support in Physically Network-Constrained Energy Systems

Shafaa mahmood shnawa
article en

Abstract

Benchmarks of inverter-based power grids that use machine learning (ML), large language models (LLMs), and learning-based inverter control can be misleading when fluent responses are operationally infeasible, judge-based scores are biased, or forecast-accuracy metrics such as Mean Absolute Error (MAE) do not reflect reserve and dispatch consequences. This study introduces VERITAS-GridBench, an operationally grounded benchmark in which decision-support outputs are evaluated against simulator/optimizer-based "Executable Truth" using explicit feasibility and constraint-violation measures. The framework combines tool verification, rule-based checks, and calibrated multi-judge semantic assessment through Triangulated Scoring, while also incorporating reproducibility, contamination control, domain-shift testing, and a Forecast-to-Operations (F2O) pathway. The experiments compare judge-only, multi-judge, and triangulated evaluation for power-system decision support and compare representative statistical, machine-learning, sequence, convolutional, and transformer forecasting models after their forecasts are propagated to reserve and dispatch calculations. The results show that judge-only scoring can reward fluent but infeasible actions, whereas tool-verified triangulation markedly increases feasibility and reduces unsafe wins and constraint violations. The forecasting results further show that lower MAE does not necessarily imply lower operational cost or risk. These findings support end-to-end evaluation in which model quality is judged by both predictive performance and physically executable grid outcomes.

Recent Innovations in Mechatronics
Thi Qar University (IQ)
Affordable and clean energy, Industry, innovation and infrastructure
Smart Grid and Power Systems
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon