Multimodal Representation Learning by Hybrid Transformer with Multi-level Fusion

Multimodal learning, which integrates data from various sensory sources, is essential for advancing artificial intelligence. However, current methods often face issues where certain modalities dominate others, leading to reduced effectiveness. In this paper, we introduce ML-Former, a new hybrid transformer architecture designed to harmonize representation learning across multiple modalities. ML-Former employs a unified input system that reduces interference between modalities, ensuring balanced contributions. It also enables deep interaction and perceptual learning to enhance the integration of diverse modalities, resulting in a more robust and nuanced representation. Extensive experiments on five diverse datasets show that ML-Former outperforms existing methods.