Architecture Design: From Neural Networks to Foundation Models

Historically, we are taught to use task-dependent architecture design and objectives to tackle data science tasks. Counter intuitively, this dogma has been proven (partly) wrong by deep neural networks, while rising into stardom, e.g., large video/language models, widely used for natural language, image, speech, and other multi-modal tasks with unprecedented performance across data science. Key attributes of current foundation models is their ability to fuse different information (potentially across different domains, e.g., text and images), and their ability to capture high-order interactions of the elements using the Transformer architecture [1]. In this tutorial, we will focus on architecture design and explain how the transition from older ideas of task-dependent architectures to large, pretrained foundation models emerged. We will also identify fundamental flaws in the computational efficiency and study the impact of architectural choices into generalization, robustness, privacy, and fairness.