Talk, Touch, See, Explore: A Multimodal Virtual Environment for Digital Thread Analytics with a Conversational Generative AI Assistant

The design of complex engineered systems relies on actively managing a linked set of Model-Based Systems Engineering (MBSE) digital artefacts across the system life cycle. However, the lack of integration between CAD models for geometry and SysML-like models for systems, together with the effort required to navigate multiple layers of MBSE data, selectively view information over time, and mentally synthesise it, hinders engineers from ensuring complete and correct digital threads. This paper presents a multimodal virtual environment that supports digital thread analytics through visualisation, direct interaction, and speech, augmented by a conversational generative AI assistant (CG-AI). Validation focused on the usability of the CG-AI in combination with 3D visualisation. Preliminary results show that users rated the assistant’s usability positively and found it helpful in explaining domainspecific knowledge and detecting potential inconsistencies or omissions. The main limitation identified was its verbosity, which increased cognitive load. Overall, participants preferred to begin exploration through visualisation and used the assistant on demand for detailed information. When relying exclusively on the assistant for task answers, users often became confused due to a lack of broader contextual understanding, prompting a return to visual exploration. Future work includes enhancing the assistant with adjustable parameters (e.g., verbosity), tailoring behaviours to MBSE tasks and user profiles (e.g., integrating metacognitive support for novices). Longer-term directions include capturing intention cues (e.g., gaze path, fixation point, hand and body movements, proximity of objects, pointing) and developing context-aware reactions. The utility and usability of the environment will be continuously assessed with systems engineers.