Optimising multi-modal fusion with a tri-encoder tensor network for process applications

Multi-modal fusion combines information from diverse modalities, enabling scalable predictions in complex data environments. To address the existing limitations in capturing modalities interactions and reducing time consumption, we propose an optimising multi-modal fusion with a tri-encoder tensor network for process applications (TS-OMMF). Specifically, the input modalities are encoded to abstract the intra-modal feature and obtain the representation respectively. The representations are fused into a high dimension space with a low-rank factor to limit the dimension, and a linear extension pattern is employed to assist model extend for multi-modal. The fusion features are input into a novel tri-encoder to capture the inter-modal feature at a finer granularity and obtain the complementary features, thereby reducing the time consumption. Extensive experiments on benchmark multi-modal datasets demonstrate that TS-OMMF improves performance metrics by 0.8% to 6.1%. These results highlight TS-OMMF practical applicability, scalability, and potential for advancing process modelling and other complex multi-modal data-driven tasks.