A Large Language Model-Based Manufacturing Execution System for Heat Treatment and Its Evaluation Method

Quality investigations in heat treatment require inspection results to be linked with processing histories, while conventional manufacturing execution systems distribute these records across separate modules. This study develops a read-only language-model interface with product and basket-time linkage and evaluates query execution separately from evidence correctness. A same-model comparison using GPT–OSS 20B covers 48 tasks and 288 attempted executions, with 273 complete responses. Frozen condition-screen counts were similar; asymmetrically reviewed scope corrections are not used as comparative evidence. Independent backend checks identified alarm-count inconsistencies before evidence reached the model. A retrospective, non-blinded review of 41 numerical fallbacks, assisted by artificial intelligence (AI), found six pre-fallback answers supported relative to the saved evidence, 34 containing unsupported or incorrect claims, and one indeterminate. These assessments do not independently validate the enterprise records. Three quality cases illustrate linkage to stage-specific furnace evidence, but incomplete basket coverage leaves the relationship between temperature deviations and hardness nonconformity undetermined. The study separates returned states, answer support, and independently verified facts when evaluating language-model access to manufacturing records.