Multimodal Digital Twin Architecture for Patient Monitoring in Art Therapy

Abstract Art therapy is a promising non-pharmacological treatment to enhance emotional well-being and psychological rehabilitation by means of engaging in activities like drawing, painting and computer interaction as a form of creative expression. Nevertheless, traditional art therapy assessment is mainly based on observation and subjective assessment by the therapist which restricts the possibility of constant monitoring of patient emotional and behavioral reactions in real-time. In a bid to overcome these drawbacks, this paper suggests a multimodal digital twin architecture to adaptive art therapy monitoring. The suggested architecture combines facial expressions, speech cues, physiological readings, and interaction with behavior to build a single, intelligent healthcare monitoring system to conduct continuous analysis of the patient-state and predictive adaptive therapy responses. The multimodal representations are first obtained by applying ConvNeXt V2, Wav2Vec2.0, and Temporal Convolution Networks (TCN). Next, a Mamba State Space Model is used to learn temporal behavior, and a cross-modal attention mechanism implemented using Q-Formers is used to create integrated contextual representations of patients. A Neural Ordinary Differential Equation (Neural ODE)-based synchronization system is used to keep the digital twin of the patient up-to-date in real time, and a Mixture-of-Experts prediction model is used to predict the treatment effectiveness and produce adaptive treatment prescriptions. Experimental results have shown the proposed framework to perform better in a variety of evaluation tasks, including a temporal emotional prediction accuracy of 95.1% and a general therapy-response prediction accuracy of 97.1%, and significantly lowering the latency of computation than traditional transformer-based methods. The proposed scheme offers a smart and responsive healthcare surveillance system to next-generation digital therapeutic setup.