Scene Text Detection Method Based on Multimodal and Multi-Task Optimization

Natural scene text detection faces multiple challenges, including complex backgrounds, diverse fonts, and irregular layouts. Although multimodal information fusion has significant advantages in enhancing local perception capabilities, text detection often encounters issues such as feature misalignment and information redundancy in practical applications. To address these challenges, this paper proposes a novel multimodal text detection model. The model transforms the embeddings generated by the image encoder into a textual form prompt and collaborates with the text encoder, effectively improving the alignment and fusion of text and image features. Additionally, a small amount of extra embeddings is incorporated, allowing the model to learn more useful global information from redundant features, further strengthening feature representation. Moreover, the model employs a multi-task learning framework, enabling simultaneous output of character-level maps, segmentation maps, and distance maps, thus enhancing the model's understanding of the structural features of text regions. Experimental results show that the proposed model performs excellently in natural scene text detection tasks, significantly improving detection accuracy and robustness for complex scenes and irregular text.