Leren van Representaties voor Visie en Taal

Representations play a crucial role in how both humans and artificial models make predictions about the world; the way information is structured can significantly impact how hard it is to make accurate predictions. In machine learning, the ability to learn useful representations automatically is a key factor behind the success of deep neural networks, which have shown remarkable capabilities for tasks involving text, images, video, and many other modalities. Despite these successes, many challenges remain. This thesis focuses on two significant challenges in machine learning: improving out-of-distribution prediction performance on the one hand, and incorporating high-level planning into language models on the other hand. It addresses these challenges by learning representations that exploit specific assumptions associated with each challenge. For the first challenge, we aim to improve out-of-distribution prediction performance by leveraging causality. Specifically, we explore the Sparse Mechanism Shift assumption, which posits that distributional changes are often due to alterations in a small subset of underlying causal mechanisms. We investigate two research questions in this context. First, we examine the effectiveness of an established technique consisting of deconfounding using automatically identified confounders in vision-language tasks. Our critical analysis reveals that automatic deconfounding does not consistently improve out-of-distribution adaptation and requires additional assumptions to align with theoretical principles. Moreover, empirical evaluations using human-labeled causal relations indicate no positive correlation between accurate causal structure identification and enhanced out-of-distribution performance. Second, we investigate whether exploiting the Sparse Mechanism Shift assumption by disentangling model parameters with respect to causal mechanisms can improve out-of-distribution adaptation. This is motivated by the idea that with successful parameter disentanglement, a model can adapt to a shifted causal mechanism by updating only the corresponding subset of its parameters. Utilizing a model that leverages interventions to causal variables, we conduct experiments on a synthetic dataset designed to simulate sparse mechanism shifts. Our results show a positive correlation between disentanglement and out-of-distribution performance; however, our method does not consistently improve performance across all models and datasets. We identify potential reasons, including limitations in improving disentanglement over baselines and unintended specialization of model parameters. For the second challenge, we aim to enhance language models by integrating abstract planning into text generation. While language models generate text sequentially without high-level planning, humans typically plan their writing at an abstract level before composing text. Inspired by this, we propose an external planner that predicts abstract latent plans to guide the language model. We ensure scalable planner training by generating supervision through clustering techniques applied to unlabeled text data. Our experiments demonstrate that incorporating the planner improves perplexity and various generation metrics, and that an external planner outperforms using the language model itself as a planner. Finally, we address the challenge of enabling joint end-to-end training of the planner and the language model. We introduce a soft selection mechanism that uses the planner's predicted probabilities to compute a mixture of plan embeddings, enabling effective end-to-end training. We find it crucial to prevent the planner from forgetting its high-level pretraining objective, employing strategies like gradual unfreezing and combining low-level and high-level objectives during fine-tuning. Our findings highlight a trade-off between perplexity and plan adherence, which can be balanced by training the language model with more oracle plans or more planner-predicted plans. In conclusion, this thesis provides new insights into leveraging causal structures for out-of-distribution performance improvement and integrating abstract planning into language models. While challenges remain, our work lays the groundwork for future research, with potential applications in developing more robust and capable machine learning models.