On the Use of Large Language Models for Inter-Model Consistency Specification

Cyber-physical systems are developed using heterogeneous modeling frameworks that address different engineering concerns and abstraction levels. In practice, models from mechanical, electrical, and software engineering coexist across development phases, creating the need to specify inter-model consistency despite differences in terminology, structure, and execution semantics. A first step of such a specification is to identify which modeling constructs of two frameworks correspond, that is, whose instances have to be kept consistent, and along which dimensions. In this paper, we investigate how well Large Language Models (LLMs) perform this correspondence identification for pairs of modeling constructs, compared with expert references. We use two gold-standard datasets: 90 automotive construct pairs across AUTOSAR, ROS 2, CAD, Simulink, and a brake specification metamodel, annotated by three experts, and 32 smart-grid construct pairs across three IEC metamodels, whose reference labels are derived from a published rule-based integration. Each pair is labeled on an ordinal scale along three dimensions: syntactic, semantic, and behavioral. Six large language models from OpenAI and Google are evaluated under zero-shot, one-shot, and few-shot prompting, together with two agentic systems, OpenAI Codex and Anthropic Claude Code, run with web access allowed or disabled. We report Quadratic Weighted Kappa, Mean Absolute Error, accuracy, and macro-F1 with bootstrap confidence intervals and paired tests. On average, the LLMs agree with the references beyond chance and beyond string- and embedding-based baselines, but ordinal agreement remains weak to moderate. On the automotive pairs with traceable independent annotations, the agreement of the best configurations with individual experts does not differ significantly from the agreement among the experts, whereas most configurations agree less in the semantic dimension and, on the AUTOSAR–ROS 2 pairs, also in the syntactic dimension. Over both datasets, semantic correspondence is judged more reliably than syntactic correspondence. Vendor differences are small and dimension-specific, and in-context examples improve agreement only in selected dimensions. On the automotive pairs, the ordinal agreement of the agentic systems does not exceed that of the best direct configurations. On the smart-grid pairs, for which their prompts, unlike the direct prompts, define the dimensions in the terms used to derive the reference labels, they reach higher syntactic agreement. Web access does not improve their agreement consistently. Overall, LLMs can support experts in identifying correspondences, but they do not replace expert judgment.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-16
DOI
https://doi.org/10.5281/zenodo.24283167
Primary Topic
Model-Driven Software Engineering Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

On the Use of Large Language Models for Inter-Model Consistency Specification

Benjamin Arp
4 links
Zenodo (CERN European Organization for Nuclear Research)
Model-Driven Software Engineering Techniques
article

On the Use of Large Language Models for Inter-Model Consistency Specification

Benjamin Arp
article en

Abstract

Cyber-physical systems are developed using heterogeneous modeling frameworks that address different engineering concerns and abstraction levels. In practice, models from mechanical, electrical, and software engineering coexist across development phases, creating the need to specify inter-model consistency despite differences in terminology, structure, and execution semantics. A first step of such a specification is to identify which modeling constructs of two frameworks correspond, that is, whose instances have to be kept consistent, and along which dimensions. In this paper, we investigate how well Large Language Models (LLMs) perform this correspondence identification for pairs of modeling constructs, compared with expert references. We use two gold-standard datasets: 90 automotive construct pairs across AUTOSAR, ROS 2, CAD, Simulink, and a brake specification metamodel, annotated by three experts, and 32 smart-grid construct pairs across three IEC metamodels, whose reference labels are derived from a published rule-based integration. Each pair is labeled on an ordinal scale along three dimensions: syntactic, semantic, and behavioral. Six large language models from OpenAI and Google are evaluated under zero-shot, one-shot, and few-shot prompting, together with two agentic systems, OpenAI Codex and Anthropic Claude Code, run with web access allowed or disabled. We report Quadratic Weighted Kappa, Mean Absolute Error, accuracy, and macro-F1 with bootstrap confidence intervals and paired tests. On average, the LLMs agree with the references beyond chance and beyond string- and embedding-based baselines, but ordinal agreement remains weak to moderate. On the automotive pairs with traceable independent annotations, the agreement of the best configurations with individual experts does not differ significantly from the agreement among the experts, whereas most configurations agree less in the semantic dimension and, on the AUTOSAR–ROS 2 pairs, also in the syntactic dimension. Over both datasets, semantic correspondence is judged more reliably than syntactic correspondence. Vendor differences are small and dimension-specific, and in-context examples improve agreement only in selected dimensions. On the automotive pairs, the ordinal agreement of the agentic systems does not exceed that of the best direct configurations. On the smart-grid pairs, for which their prompts, unlike the direct prompts, define the dimensions in the terms used to derive the reference labels, they reach higher syntactic agreement. Web access does not improve their agreement consistently. Overall, LLMs can support experts in identifying correspondences, but they do not replace expert judgment.

Zenodo (CERN European Organization for Nuclear Research)
Karlsruhe Institute of Technology (DE)
Model-Driven Software Engineering Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon