Dual-channel representation fusion and structural information for framework recognition

To address the issues of single representation and insufficient utilization of structure in existing framework recognition methods under complex contexts, a novel end-to-end recognition framework driven by dual-channel representation fusion and structural information collaboration is proposed. This method first builds a visualsemantic dual-channel encoder: the visual channel simultaneously extracts the spatial position, shape, and scale features of framework elements through object detection and instance segmentation. The semantic channel captures context word-level and sentence-level semantic embeddings using a pre-trained language model and enhances the understanding of abstract roles and implicit relationships through an attention mechanism. To avoid the differences between heterogeneous modalities, a cross-modal gated fusion module is designed to adaptively calibrate the weights of the two channels, achieving complementary enhancement. Secondly, a structure-aware graph convolutional network is introduced, modeling candidate elements and context words as nodes, and constructing edges based on dependency syntax, co-occurrence statistics, and common sense associations. It iteratively propagates topological priors, suppresses redundant nodes, and highlights key paths. Finally, the fused features are simultaneously output as framework categories and element roles through a lightweight decoder. The entire network can be trained end-to-end without the need for manual templates. Experiments show that this method achieves significant improvements on multiple datasets in fields such as event extraction and script understanding, verifying the effectiveness of dual-channel fusion and structural information collaboration, and demonstrating good interpretability and scalability.