Context aware hierarchical alignment for robust multimodal three stream sentiment analysis

Lack of progress in sentiment analysis for conversational text because desktop and mobile communications shift emotions irregularly between different channels. Traditional approaches do not achieve an effective understanding of genuine situations when performing healthcare monitoring, providing customer support, and social media evaluations because they inadequately combine visual audio and text data. The research outlines CHARM3S which offers sentiment analysis capabilities by building hierarchical structures that analyze contextual sentiments from all communication channels. Thus, the new proposed system combines temporal dependency maintenance through new context-aware pooling with solutions for modality alignment through orthogonality constraints together with central moment discrepancy regularization. The model includes three specific encoders where DeBERTa-v3 starts the text processing before Data2Vec processes audio and facial expressions using vision transformers with bidirectional cross-modal attention for enhanced multichannel representation learning. The CHARM3S system outperforms comparable methods during the analysis phase of emotional phone calls between agents and customers, telehealth sessions, and interactive conversations between agents and customers. And the new implementation of this system exists through uncertainty-based methodologies, which ensure operational effectiveness with untrustworthy or incomplete modality information. The Codebase and implementation is made available at https://github.com/ManojPennada/CHARM3S.