3D Visual Question Answering via 2D-Enhanced Multi-Modal Fusion Framework

3D Visual Question Answering (3D-VQA), which focuses on answering user questions based on a given 3D scene, has attracted increasing attention from researchers. As far as we know, most existing methods first encode 3D point clouds and text separately, then align their features through a fusion layer, and ultimately predict answers based on the fused representations. However, the cross-modal alignment during the fusion stage remains suboptimal, primarily due to the sparsity of 3D point clouds and the lack of 3D scene-text pairs. To address this issue, we propose a 2D-enhanced multi-modal fusion framework for 3D-VQA, leveraging 2D information to supplement the contextual and detailed information missing from the point clouds, thereby enhancing the model's cross-modal alignment capabilities. Specifically, the proposed model consists of three stages. First, it employs a 2D visual-language model to retrieve scene images relevent to the question as supplementary information. Next, a Dual-branch Fusion Module is designed to facilitate bidirectional interaction between the 2D and 3D modalities, enhancing the integration and alignment of visual and textual features. Finally, the fused features are decoded to generate answers. Experimental results demonstrate that our method outperforms existing state-of-the-art baselines.