Mediafusionnet for automated digital media content recognition using multimodal deep learning

With the explosive growth of short-video platforms, live streaming, and social media, automatically recognizing and classifying digital media content with high efficiency and accuracy has become a core task in information retrieval, content moderation, and recommendation systems. However, existing methods predominantly rely on single-modality features, which fail to capture the rich semantic information embedded in media content and lack adaptive modeling of modality-contribution differences across diverse content types, resulting in significantly limited recognition accuracy in heterogeneous content scenarios. To address these challenges, this paper proposes MediaFusionNet , a novel multimodal deep learning model that achieves hierarchical fine-grained semantic interaction and content-adaptive deep fusion of textual, acoustic, and visual modalities within a unified framework. The proposed Hierarchical Cross-Modal Attention (HCMA) module performs pairwise multi-head attention computation across all three modality pairs and aggregates cross-modal signals via learnable weights, capturing fine-grained cross-modal interactions followed by target-specific multi-source semantic aggregation. The Adaptive Gated Fusion Network (AGFN) dynamically allocates per-modality fusion weights conditioned on the semantic characteristics of each input sample via Sigmoid scoring and Softmax temperature normalization, supplemented by a GELU residual bypass to compensate for information loss. We organized the Multimodal Media Content (MMC) dataset from public multimodal sources using existing labels or metadata mappings, comprising 27,640 annotated video segments across five content categories (News, Entertainment, Education, Sports, Lifestyle). Comprehensive experiments against seven baseline methods on MMC demonstrate that MediaFusionNet achieves the best performance among the compared methods across five metrics—Accuracy (87.3%), Macro-F1 (86.1%), Precision (86.7%), Recall (86.4%), and AUC (0.934)—surpassing the strongest compared baseline by 2.4 percentage points in accuracy. Ablation studies, controlled fusion-strategy comparisons, representation-transfer evaluation, computational cost analysis, and modality-weight analysis further validate the effectiveness of each module and the overall robustness of the proposed approach. This work provides a new technical pathway for multimodal media content understanding, with significant academic and practical implications.

Authors

Institutions

Publication Details

Journal
Discover Artificial Intelligence
Published
2026-11-20
DOI
https://doi.org/10.1007/s44163-026-02779-8
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Mediafusionnet for automated digital media content recognition using multimodal deep learning

Long Wang, Yu Jiang, Hailong Li
1 links
Discover Artificial Intelligence
Multimodal Machine Learning Applications
article

Mediafusionnet for automated digital media content recognition using multimodal deep learning

Long Wang, Yu Jiang, Hailong Li
article en

Abstract

With the explosive growth of short-video platforms, live streaming, and social media, automatically recognizing and classifying digital media content with high efficiency and accuracy has become a core task in information retrieval, content moderation, and recommendation systems. However, existing methods predominantly rely on single-modality features, which fail to capture the rich semantic information embedded in media content and lack adaptive modeling of modality-contribution differences across diverse content types, resulting in significantly limited recognition accuracy in heterogeneous content scenarios. To address these challenges, this paper proposes MediaFusionNet , a novel multimodal deep learning model that achieves hierarchical fine-grained semantic interaction and content-adaptive deep fusion of textual, acoustic, and visual modalities within a unified framework. The proposed Hierarchical Cross-Modal Attention (HCMA) module performs pairwise multi-head attention computation across all three modality pairs and aggregates cross-modal signals via learnable weights, capturing fine-grained cross-modal interactions followed by target-specific multi-source semantic aggregation. The Adaptive Gated Fusion Network (AGFN) dynamically allocates per-modality fusion weights conditioned on the semantic characteristics of each input sample via Sigmoid scoring and Softmax temperature normalization, supplemented by a GELU residual bypass to compensate for information loss. We organized the Multimodal Media Content (MMC) dataset from public multimodal sources using existing labels or metadata mappings, comprising 27,640 annotated video segments across five content categories (News, Entertainment, Education, Sports, Lifestyle). Comprehensive experiments against seven baseline methods on MMC demonstrate that MediaFusionNet achieves the best performance among the compared methods across five metrics—Accuracy (87.3%), Macro-F1 (86.1%), Precision (86.7%), Recall (86.4%), and AUC (0.934)—surpassing the strongest compared baseline by 2.4 percentage points in accuracy. Ablation studies, controlled fusion-strategy comparisons, representation-transfer evaluation, computational cost analysis, and modality-weight analysis further validate the effectiveness of each module and the overall robustness of the proposed approach. This work provides a new technical pathway for multimodal media content understanding, with significant academic and practical implications.

Discover Artificial IntelligenceVol. 6(1)
Shandong Huayu University of Technology (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon