Formal Modeling of Augmented Reality Maintenance-Guidance Processes Based on Multimodal Knowledge Extraction and Graph Reasoning

Augmented reality maintenance guidance (ARMG) provides support for industrial maintenance by converting knowledge into instructions and integrating virtual information with physical equipment. However, constructing guidance processes still depends on the manual interpretation of heterogeneous maintenance materials, while inconsistent knowledge representations and missing information hinder automated modeling. This study proposes a formal modeling method that integrates multimodal knowledge extraction with graph reasoning. A Maintenance Guidance Ontology (MGO) and a six-tuple maintenance-event template are first developed to unify the semantic representation of actions, objects, resources, and related elements. Qwen3-VL is then fine-tuned to the ARMG domain through low-rank adaptation (LoRA) to extract maintenance knowledge from text and video, respectively. Finally, for missing elements in extracted knowledge, we develop an MGO-constrained semantic–temporal graph-convolutional network (MGO-STGCN). It combines relation semantics, step order, weighted neighborhood aggregation, and an ontology-constrained mask. The model predicts and completes three missing-element types: tools, spare parts and consumables, and precautions. Both extraction models outperformed the comparison methods in terms of slot-level and event-level F1 scores. MGO-STGCN achieved F1 scores of 0.799, 0.735 and 0.680 for the three completion tasks. Ablation experiments indicated that relation–temporal weighting and ontological constraints make complementary contributions to completion performance. These results demonstrate the potential of the proposed framework for unified formal representation of multimodal maintenance knowledge and completion of missing guidance information.