Generalization Boost in Bimodal Classification via Data Fusion Trained on Sparse Datasets

For internet users it is frequently challenging to properly navigate and interpret the overwhelming amount of information available to them. Multimodal artificial intelligence (AI) systems can aid in the process; for example, by helping to debunk misinformation, or by placing video content into a proper context. The training of commensurate AI systems requires the labelling of sizable multimodal datasets, a process that is very time-consuming and prone to being outpaced by the rapid changes of the information landscape. To address this problem, we are introducing a hybrid fusion algorithm, Bimodal Gaussian-Logits glObal stream Weighting (BiGLOW), which performs effective information integration in scenarios in which only a very limited amount of bimodally labelled training data is available. The method is motivated from Bayesian inference; we model a modality-weighted linear combination of differences in logits between class pairs through a Gaussian distribution. We evaluated the proposed method with two bimodal multiclass datasets: the Fakeddit dataset for fake news classification and the TAU Audio-Visual Urban Scenes 2021 for urban scene identification. The two datasets capture four of the most frequently used modalities on the web: text, images, audio, and video. Scenarios with scarce training data were created by reducing the respective size of the bimodally labelled set to various small fractions of the overall data size. Our experimental results indicate that the proposed hybrid fusion algorithm surpasses competing purely neural-network-based models in terms of the model complexity, average accuracy, robustness, and time requirements for both model training and evaluation. We attribute the improved performance of the proposed method to its implied, Bayesian inference inspired regularization.