Multimodal integration in machine learning

Multimodal integration, a key aspect of human cognition, combines information from various sensory modalities. It has been a focus of development in machine learning for many decades. It continues to be a growing area of interest due to its potential benefits for accuracy, robustness, and reliability. However, finding an optimal fusion strategy is not easy, as it can vary depending on the task and potentially even the data. This thesis explores advanced fusion strategies, focusing on audio-visual speech recognition (AVSR), the analysis of social media data regarding hate speech and misogyny, and multimodal information integration on sparse datasets. In the AVSR study, we developed a Decision Fusion Network (DFN) that combines the concepts of representation fusion and decision fusion. A broad range of reliability measures were extracted to dynamically inform the fusion model about the degree of reliability of each modality. Our experiments covered both conventional hybrid and state-of-the-art end-to-end (E2E) speech recognition systems. The proposed fusion model achieved outstanding recognition rates and robustness across various simulated noise environments. In the second study on hate speech detection, we initially explored text-only hate speech detection and found that data-driven features like BERT embeddings are more suitable for this task compared to hand-crafted features like term frequency-inverse document frequency (TF-IDF). Our proposed text-only model, utilizing BERTweet embeddings for identifying spreaders of irony, won the PAN-2022 challenge. For detecting hateful memes, our previous models were submitted to the MAMI and Memotion 3.0 challenges and won two of the three tasks of the Memotion challenge. The newly proposed models employed a pre-trained CLIP model to extract data-driven features for fusion. Notably, the newly proposed Hate-Attention model, which is simple yet effective, demonstrated superior performance compared to state-of-the-art models and even outperformed large language models like ChatGPT 4.o and TinyLLaVA. In both considered application areas, we found that neural network-based fusion models are prone to overfitting, require large amounts of training data and are very time-consuming to train. If only sparse training sets are available, this leads to serious overfitting and generalization issues. To address this issue, we proposed two fusion algorithms: Gaussian Logits glObal stream Weighting (GLOW) for binary classification and pairwise Gaussian Logits glObal stream Weighting (BiGLOW) for multiclass classification. These fusion strategies, designed for Bayes-optimal decision making, demonstrate superior performance in terms of accuracy, robustness, processing time, number of parameters, and reliability, particularly on sparse training sets. The insights gained from this thesis contribute to the development of more efficient, robust, and adaptable multimodal models, with implications for future advancements in speech recognition, content moderation, and beyond. More importantly, the ability to train models effectively with sparse training samples, fewer parameters, and within shorter time-frames, presents significant advantages for deploying artificial intelligence in resource-constrained environments.