MBBo-RPSLD: Training a Multimodal BlenderBot for Rehabilitation in Post-Stroke Language Disorder

Stroke, a severe cerebrovascular event, can lead to motor deficits and often impairs language, affecting quality of life. Thus, developing effective rehabilitation models is crucial for enhancing language function and well-being in stroke patients. This paper presents the Multi-Blender model, designed to address the challenges of multimodal data processing and the complexity of medical dialogue in stroke language rehabilitation. The model integrates the multimodal encoding capabilities of ImageBind-LLM with the conversational generation strengths of BlenderBot, creating a tailored rehabilitation solution for stroke patients. We evaluated the model using a range of datasets, including the NINDS dataset, MSDM database, and clinical data from hospitals, focusing on audio-video recognition and speech translation tasks. Our results demonstrate that the Multi-Blender model outperforms existing models, achieving a BLEU score of 30.2 in the AST task, surpassing Whisper Large-v2 and AudioPaLM. In the ASR task, it also displayed superior performance. The model's effectiveness was further validated through an adjusted MME benchmark, where it scored 85.25% in perceptual tasks and 76.83% in cognitive tasks, outperforming other models in language understanding and fluency scoring. These findings indicate that the Multi-Blender model significantly enhances stroke language rehabilitation by improving multimodal data processing and providing accurate, reliable solutions. Future work will focus on expanding the training dataset and optimizing the model to further advance the effectiveness of stroke rehabilitation.