Scenario-driven multimodal dialogue generation method for reminiscence therapy

针对现有对话生成技术在回忆疗法中因缺乏专用高质量语料导致交互生硬、引导效果不佳的问题，提出一种情景 驱动的角色信息生成与角色扮演(SCIGR)多轮对话数据构建方法。首先，为图片生成详细描述以构建回忆情景；其次，融合检 索增强生成技术与提示工程，创作贴近老年人生活背景的角色信息；再次，构建图片背后的拍摄故事以丰富对话上下文；最 后，基于上述信息采用角色扮演技术生成多轮对话，构建首个中文图片回忆疗法多轮对话数据集 IRTDialogues。为验证该数据 集的有效性，采用多模态大语言模型(MLLM)进行监督微调(SFT)，并构建包含共情性、引导性等 5 个维度的多维多模态回忆 疗法评测基准(MMRT)进行评估。实验结果表明，基于 IRTDialogues 数据集训练的模型能生成更符合情境、更具情感支持性的 回应，为构建更具人文关怀的智能养老技术提供了数据支持和新的实现路径。Since existing dialogue generation technologies in reminiscence therapy suffer from rigid interactions and poor guidance due to the lack of high-quality domain-specific corpora, a Scenario-driven Character Information Generation and Role-playing (SCIGR) approach was proposed to generate character information and construct multi-turn role-playing dialogue data. First, detailed image descriptions were produced to create reminiscence scenarios. Then, retrieval-augmented generation techniques and prompt engineering were integrated to develop character information that aligns with the life backgrounds of elderly individuals. Subsequently, photo-shooting stories were constructed to enrich the dialogue context. Finally, multi-turn dialogues were generated through role-playing based on the above information, resulting in the establishment of the first Chinese multi-turn dialogue dataset for image-based reminiscence therapy, named IRTDialogues. To verify the effectiveness of this dataset, a Multimodal Large Language Model (MLLM) was trained using Supervised Fine-Tuning (SFT), and a Multi-dimensional Multimodal Reminiscence Therapy benchmark (MMRT) covering five dimensions—including empathy and guidance—was developed for evaluation. Experimental results demonstrate that models train on the IRTDialogues dataset are capable of generating responses that are more contextually appropriate and emotionally supportive. This provides both data support and a new implementation pathway for developing intelligent elderly care technologies with greater humanistic concern