Bootstrapping OTS-Funcimg pre-training model (Botfip): a comprehensive multimodal scientific computing framework and its application in symbolic regression task

In the realm of scientific computing, many problem-solving approaches focus primarily on processes and outcomes. Even in AI applications within science, a notable absence of deep multimodal information mining is often observed, with a lack of frameworks analogous to those in the image-text domain. This paper introduces a novel scientific computing multimodal framework based on Function Images (Funcimg) and Operation Tree Skeleton Sequence (OTS), named Bootstrapping OTS-Funcimg Pre-training Model (Botfip), which is inspired by the BLIP model from the image-text field. Botfip employs image encoders such as ViT and sequence encoders like BERT, aligning these encoders during the pre-training phase by applying contrastive learning on a large-scale dataset of Funcimg-OTS pairs. This approach successfully facilitates the multimodal information mining of functions, serving as the foundation for completing corresponding downstream tasks such as symbolic regression (SR). Experiments in this paper demonstrate Botfip’s exceptional capability to mine multimodal symbolic and numerical information during the pre-training phase and highlight its performance in SR tasks, especially in tackling low-complexity SR problems. As a Multimodal framework, Botfip shows promising potential for future applications across a broader spectrum of scientific computing challenges.