Multimodal Feature Fusion and Enhancement for Function Graph Data

Recent years have witnessed performance improvements in Multimodal Large Language Models (MLLMs) on downstream natural image understanding tasks. However, when applied to the function graph reasoning task, which is highly information-dense and abundant in fine-grained structural details, these models face pronounced performance degradation. The challenges are primarily characterized by several core issues: the static projection bottleneck, inadequate cross-modal interaction, and insufficient visual context in text embeddings. To address these problems, this study proposes a multimodal feature fusion enhancement method for function graph reasoning and constructs the FuncFusion-Math model. The core innovation of this model resides in its design of a dual-path feature fusion mechanism for both image and text. Specifically, the image fusion module adopts cross-attention and self-attention mechanisms to optimize visual feature representations under the guidance of textual semantics, effectively mitigating fine-grained information loss. The text fusion module, through feature concatenation and Transformer encoding layers, deeply integrates structured mathematical information from the image into the textual embedding space, significantly reducing semantic deviation. Furthermore, this study utilizes a four-stage progressive training strategy and incorporates the LoRA technique for parameter-efficient optimization. Experimental results demonstrate that the FuncFusion-Math model, with 3B parameters, achieves an accuracy of 43.58% on the FunctionQA subset of the MathVista test set, outperforming a 7B-scale baseline model by 13.15%, which validates the feasibility and effectiveness of the proposed method.