Curriculum trustworthy multi-modal learning

Trustworthy multi-modal learning reliably integrates multiple data sources. However, current methods often face a challenge, which is the inherent non-convex nature of deep neural networks. It leads to their susceptibility to local minima, ultimately resulting in a reduced capacity for generalization. To address this issue, we first create a theoretical framework, which extends the application of curriculum learning in multi-modal scenarios. Secondly, we propose a novel curriculum termed the Dynamic SRM Curriculum (DSRMC). It consists of two modules: a scoring function and a training schedule. The scoring function sorts samples from simple to complex. The training scheduler aims to manage the quantity of samples supplied at each round during training. DSRMC facilitates positioning the learned model in a flatter region of the loss landscape, thereby enhancing its overall generalization ability. Building on DSRMC, we eventually propose an innovative method termed as Curriculum Trustworthy Multi-modal Learning (CTML). It applies DSRMC in multi-modal learning application scenarios. Extensive experiments conducted on three open datasets show that the proposed CTML outperforms state-of-the-art methods, with a maximum improvement of 6.7% in macro F1 score. Our code and dataset are publicly available on https://github.com/HackerHyper/DSRMC.git .