Enhanced FusionNet: a deep learning structure for multimodal input-based emotion-aware deepfake speech identification

Voice-based deepfakes pose a significant security concern for voice-based systems and require effective methods for distinguishing genuine and synthetic speech. This study presents Enhanced FusionNet, an emotion-aware deep learning framework for deepfake speech detection that combines Log-Mel-based acoustic representations with MultiScaleCNN, Bidirectional Long Short-Term Memory (BiLSTM), Multi-Head Cross-Attention, Residual Feature Fusion, and Adaptive Gated Fusion. An emotion representation is learned separately from the RAVDESS dataset using a speaker-independent emotion encoder and subsequently transferred to the FoR-based deepfake detection framework. On the independent FoR hold-out test set, Enhanced FusionNet achieved 98.67% accuracy, 98.18% precision, 99.17% recall, 98.67% F1-score, and 99.88% ROC-AUC. Five-fold cross-validation produced a mean accuracy of 99.08 ± 0.35% and ROC-AUC of 99.94 ± 0.05%. The model further achieved 99.40% sensitivity, 98.75% specificity, and 0.91% EER in the out-of-fold evaluation. Component-level ablation analysis demonstrated the varying contributions of the major architectural components, while a controlled FoR-only versus FoR + Emotion experiment showed that the transferred emotion representation provided complementary information but did not improve overall detection performance under the evaluated protocol. The findings demonstrate the effectiveness of Enhanced FusionNet under the evaluated benchmark conditions, while also highlighting the need for external cross-dataset and unseen-generation evaluation. Future work will focus on multilingual and cross-domain evaluation, contemporary anti-spoofing baselines, unseen synthesis methods, and computationally efficient variants for real-time deployment.

Authors

Institutions

Publication Details

Journal
Frontiers in Artificial Intelligence
Published
2026-11-13
DOI
https://doi.org/10.3389/frai.2026.1978052
Primary Topic
Speech Recognition and Synthesis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Enhanced FusionNet: a deep learning structure for multimodal input-based emotion-aware deepfake speech identification

Anil Vithalrao Turukmane, Bhargavi Anumula
4 links
Frontiers in Artificial Intelligence
Speech Recognition and Synthesis
article

Enhanced FusionNet: a deep learning structure for multimodal input-based emotion-aware deepfake speech identification

Anil Vithalrao Turukmane, Bhargavi Anumula
article en

Abstract

Voice-based deepfakes pose a significant security concern for voice-based systems and require effective methods for distinguishing genuine and synthetic speech. This study presents Enhanced FusionNet, an emotion-aware deep learning framework for deepfake speech detection that combines Log-Mel-based acoustic representations with MultiScaleCNN, Bidirectional Long Short-Term Memory (BiLSTM), Multi-Head Cross-Attention, Residual Feature Fusion, and Adaptive Gated Fusion. An emotion representation is learned separately from the RAVDESS dataset using a speaker-independent emotion encoder and subsequently transferred to the FoR-based deepfake detection framework. On the independent FoR hold-out test set, Enhanced FusionNet achieved 98.67% accuracy, 98.18% precision, 99.17% recall, 98.67% F1-score, and 99.88% ROC-AUC. Five-fold cross-validation produced a mean accuracy of 99.08 ± 0.35% and ROC-AUC of 99.94 ± 0.05%. The model further achieved 99.40% sensitivity, 98.75% specificity, and 0.91% EER in the out-of-fold evaluation. Component-level ablation analysis demonstrated the varying contributions of the major architectural components, while a controlled FoR-only versus FoR + Emotion experiment showed that the transferred emotion representation provided complementary information but did not improve overall detection performance under the evaluated protocol. The findings demonstrate the effectiveness of Enhanced FusionNet under the evaluated benchmark conditions, while also highlighting the need for external cross-dataset and unseen-generation evaluation. Future work will focus on multilingual and cross-domain evaluation, contemporary anti-spoofing baselines, unseen synthesis methods, and computationally efficient variants for real-time deployment.

Frontiers in Artificial IntelligenceVol. 9
VIT-AP University (IN), Vellore Institute of Technology University (IN)
Speech Recognition and Synthesis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon