CogniAlign: Word-level multimodal speech alignment with gated cross-attention for Alzheimer’s detection

• Propose CogniAlign, a multimodal model for Alzheimer’s disease detection that effectively integrates audio and text modalities using attention-based mechanisms, achieving state-of-the-art performance. • Introduce a word-level alignment strategy that synchronizes audio and text embeddings, enabling more semantically grounded multimodal fusion. • Incorporate prosodic features, specifically speech pauses, to enrich the representation and improve both unimodal and multimodal performance. • Conduct a comprehensive ablation study to evaluate the impact of each modality, fusion strategy, and architectural design choice. • Perform a corpus analysis to examine the influence of the proposed prosodic features within the dataset. Additionally, we analyzed the Integrated Gradients to identify the most relevant segments of each sample that the model considers when predicting Alzheimer’s disease and cognitive health. Early detection of cognitive disorders such as Alzheimer’s disease is critical for enabling timely clinical intervention and improving patient outcomes. In this work, we introduce CogniAlign, a multimodal architecture for Alzheimer’s detection that integrates audio and textual modalities, two non-intrusive sources of information that offer complementary insights into cognitive health. Unlike prior approaches that fuse modalities at a coarse level, CogniAlign leverages a word-level temporal alignment strategy that synchronizes audio embeddings with corresponding textual tokens based on transcription timestamps. This alignment supports the development of token-level fusion techniques, enabling more precise cross-modal interactions. To fully exploit this alignment, we propose a Gated Cross-Attention Fusion mechanism, where audio features attend over textual representations, guided by the superior unimodal performance of the text modality. In addition, we incorporate prosodic cues, specifically interword pauses, by inserting pause tokens into the text and generating audio embeddings for silent intervals, further enriching both streams. We evaluate CogniAlign on the ADReSSo dataset, where it achieves an accuracy of 87.35% over a Leave-One-Subject-Out setup and of 90.36% over a 5 fold Cross-Validation, outperforming existing state-of-the-art methods. A detailed ablation study confirms the advantages of our alignment strategy, attention-based fusion, and prosodic modeling. Finally, we perform a corpus analysis to assess the impact of the proposed prosodic features and apply Integrated Gradients to identify the most influential input segments used by the model in predicting cognitive health outcomes.