A digital-driven deep learning framework for text–image multimodal machine translation with gated cross-modal fusion

Text-only neural machine translation (NMT) models can struggle with lexical, gender, and referential ambiguities when textual cues alone are insufficient. This paper presents a digital-driven deep-learning framework for text–image multimodal machine translation that integrates visual context through gated cross-modal fusion and contrastive text-image alignment. The architecture combines a frozen CLIP ViT-B/32 vision encoder with a trainable linear projection and a pretrained MarianMT text encoder-decoder. An adaptive sample-level scalar gate regulates the contribution of visual information according to cross-modal relevance. On the Multi30K EN–DE Test 2016 benchmark, the proposed framework achieves 40.9 BLEU and 0.867 COMET, improving upon text-only MarianMT by 2.2 BLEU and 0.025 COMET and outperforming the evaluated multimodal baselines across seven translation metrics. Paired bootstrap resampling with 1,000 iterations indicates statistically significant improvements ( \(p < 0.001\) , Cohen’s \(d = 0.72\) , 95% CI [1.2, 3.2]). Analysis of gate activations further examines how visual contribution varies across samples with different linguistic characteristics, without assuming token-level gating. The framework introduces an additional inference latency of 21.7 ms over the text-only baseline and is further evaluated under multiple visual degradation and image-relevance conditions. These results demonstrate the potential of adaptive gated fusion for incorporating visual evidence into multimodal machine translation while maintaining computational efficiency.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-13
DOI
https://doi.org/10.1038/s41598-026-75276-8
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A digital-driven deep learning framework for text–image multimodal machine translation with gated cross-modal fusion

Nan Li
2 links
Scientific Reports
Multimodal Machine Learning Applications
article

A digital-driven deep learning framework for text–image multimodal machine translation with gated cross-modal fusion

Nan Li
article en

Abstract

Text-only neural machine translation (NMT) models can struggle with lexical, gender, and referential ambiguities when textual cues alone are insufficient. This paper presents a digital-driven deep-learning framework for text–image multimodal machine translation that integrates visual context through gated cross-modal fusion and contrastive text-image alignment. The architecture combines a frozen CLIP ViT-B/32 vision encoder with a trainable linear projection and a pretrained MarianMT text encoder-decoder. An adaptive sample-level scalar gate regulates the contribution of visual information according to cross-modal relevance. On the Multi30K EN–DE Test 2016 benchmark, the proposed framework achieves 40.9 BLEU and 0.867 COMET, improving upon text-only MarianMT by 2.2 BLEU and 0.025 COMET and outperforming the evaluated multimodal baselines across seven translation metrics. Paired bootstrap resampling with 1,000 iterations indicates statistically significant improvements ( \(p < 0.001\) , Cohen’s \(d = 0.72\) , 95% CI [1.2, 3.2]). Analysis of gate activations further examines how visual contribution varies across samples with different linguistic characteristics, without assuming token-level gating. The framework introduces an additional inference latency of 21.7 ms over the text-only baseline and is further evaluated under multiple visual degradation and image-relevance conditions. These results demonstrate the potential of adaptive gated fusion for incorporating visual evidence into multimodal machine translation while maintaining computational efficiency.

Scientific Reports
North China University of Science and Technology (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon