Visible-Infrared Dual-Band Image Captioning via Geometric Feature Enhancement and Dynamic Fusion

Visible-infrared dual-band image captioning aims to generate accurate descriptions by exploiting complementary information from visible and infrared images. However, insufficient spatial relationship modeling and inflexible cross-modal fusion limit the effectiveness of existing methods. To address these issues, we propose a Transformer-based dual-band image captioning framework consisting of three key components. First, the Dual-band Multi-level Memory Encoder (DMME) integrates geometric and visual features to enhance spatial relationship modeling and employs learnable memory vectors for multi-level cross-modal alignment and feature fusion. Second, the Hierarchical Grid-Region Interaction (HGRI) module establishes hierarchical interactions between grid-level and region-level features, facilitating fine-grained cross-modal feature associations. Third, the Hierarchical Dynamic Fusion (HDF) decoder incorporates global visual constraints and an adaptive gating mechanism to dynamically balance the contributions of visible and infrared features across different scenes. Experiments on a visible-infrared image-captioning dataset demonstrate that our method achieves a CIDEr score of 122.9 ± 0.7, outperforming existing state-of-the-art methods.

Authors

Institutions

Publication Details

Journal
Sensors
Published
2026-11-19
DOI
https://doi.org/10.3390/s26227314
Primary Topic
Multimodal Machine Learning Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Visible-Infrared Dual-Band Image Captioning via Geometric Feature Enhancement and Dynamic Fusion

Suzhen Lin, Zanxia Jin, Jialong An
Sensors
Multimodal Machine Learning Applications
article

Visible-Infrared Dual-Band Image Captioning via Geometric Feature Enhancement and Dynamic Fusion

Suzhen Lin, Zanxia Jin, Jialong An
article en

Abstract

Visible-infrared dual-band image captioning aims to generate accurate descriptions by exploiting complementary information from visible and infrared images. However, insufficient spatial relationship modeling and inflexible cross-modal fusion limit the effectiveness of existing methods. To address these issues, we propose a Transformer-based dual-band image captioning framework consisting of three key components. First, the Dual-band Multi-level Memory Encoder (DMME) integrates geometric and visual features to enhance spatial relationship modeling and employs learnable memory vectors for multi-level cross-modal alignment and feature fusion. Second, the Hierarchical Grid-Region Interaction (HGRI) module establishes hierarchical interactions between grid-level and region-level features, facilitating fine-grained cross-modal feature associations. Third, the Hierarchical Dynamic Fusion (HDF) decoder incorporates global visual constraints and an adaptive gating mechanism to dynamically balance the contributions of visible and infrared features across different scenes. Experiments on a visible-infrared image-captioning dataset demonstrate that our method achieves a CIDEr score of 122.9 ± 0.7, outperforming existing state-of-the-art methods.

SensorsVol. 26(22)
North University of China (CN)
Multimodal Machine Learning Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon