Heterogeneous Memristive Neuromorphic Computing with Physics-Aware Cross-Modal Attention for Energy-Efficient Multimodal Emotion Recognition

Multimodal Emotion Recognition (MER) is a crucial technology for achieving emotional intelligence in human–computer interaction. However, the substantial computational demands of MER models currently overwhelm the capacities of resource-constrained edge devices. To address this problem, this paper proposes a Heterogeneous Memristive Neural Network for Multimodal Emotion Recognition. The proposed network adopts a hardware–algorithm co-design approach, in which different memristive devices are assigned to different modal processing tasks according to their physical characteristics. Specifically, highly stable filamentary memristors are utilized to construct deterministic feature extraction pathways for text and video, whereas highly plastic Phase-Change Memory (PCM) crossbar arrays are designed for audio encoding. Based on the high plasticity of PCM, a memristive cross-modal attention network is further designed to achieve dynamic and adaptive multimodal feature fusion. Furthermore, a collaborative weight update method is proposed. During weight updates, the inherent writing noise and conductance drift of PCM are utilized as hardware-level regularization and automatic sparse learning strategies. Simulation experiments conducted on the IEMOCAP and MELD datasets using the MemTorch framework demonstrate that the proposed neural network maintains recognition accuracy comparable to GPU baseline models, even when accounting for device non-idealities. Overall, the proposed neural network effectively resolves the “performance–power” conflict when deploying high-performance MER models on resource-constrained edge devices, providing an efficient and stable neuromorphic computing solution for edge applications.

Authors

Institutions

Publication Details

Journal
Electronics
Published
2026-11-12
DOI
https://doi.org/10.3390/electronics15225257
Primary Topic
Advanced Memory and Neural Computing
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Heterogeneous Memristive Neuromorphic Computing with Physics-Aware Cross-Modal Attention for Energy-Efficient Multimodal Emotion Recognition

Fei Yu, Cong Xu, Jie Jin, Wei Feng et al.
Electronics
Advanced Memory and Neural Computing
article

Heterogeneous Memristive Neuromorphic Computing with Physics-Aware Cross-Modal Attention for Energy-Efficient Multimodal Emotion Recognition

Fei Yu, Cong Xu, Jie Jin, Wei Feng, Le-Jian Xu
article en

Abstract

Multimodal Emotion Recognition (MER) is a crucial technology for achieving emotional intelligence in human–computer interaction. However, the substantial computational demands of MER models currently overwhelm the capacities of resource-constrained edge devices. To address this problem, this paper proposes a Heterogeneous Memristive Neural Network for Multimodal Emotion Recognition. The proposed network adopts a hardware–algorithm co-design approach, in which different memristive devices are assigned to different modal processing tasks according to their physical characteristics. Specifically, highly stable filamentary memristors are utilized to construct deterministic feature extraction pathways for text and video, whereas highly plastic Phase-Change Memory (PCM) crossbar arrays are designed for audio encoding. Based on the high plasticity of PCM, a memristive cross-modal attention network is further designed to achieve dynamic and adaptive multimodal feature fusion. Furthermore, a collaborative weight update method is proposed. During weight updates, the inherent writing noise and conductance drift of PCM are utilized as hardware-level regularization and automatic sparse learning strategies. Simulation experiments conducted on the IEMOCAP and MELD datasets using the MemTorch framework demonstrate that the proposed neural network maintains recognition accuracy comparable to GPU baseline models, even when accounting for device non-idealities. Overall, the proposed neural network effectively resolves the “performance–power” conflict when deploying high-performance MER models on resource-constrained edge devices, providing an efficient and stable neuromorphic computing solution for edge applications.

ElectronicsVol. 15(22)
Hunan University of Science and Technology (CN), Panzhihua University (CN), Changsha University of Science and Technology (CN)
Advanced Memory and Neural Computing
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon