Navigating data limitations in open world: exhaustiveness, balance, and correctness

The rapid development of deep learning across critical domains, ranging from autonomous transportation to healthcare diagnostics, from environmental monitoring to cybersecurity, underscores its capacity to drive societal progress. The fuel for this transformative wave is the symbiosis of a huge amount of data and computational resources. However, a significant hurdle in this evolution is the deficiency of comprehensively annotated datasets. With the advancement of Deep Learning (DL) and Machine Learning (ML) in recent years, the research focus has gradually shifted from the previously dominant closed-world scenarios to more realistic and challenging open-world scenarios. In this dissertation, I explore three of the most prevalent and significant data limitations in open world: Exhaustiveness, Balance, and Correctness. Exhaustiveness, encompassing all categories in the real world, is an inherently impossible goal given the vast expanse of human ignorance in numerous domains. Nonetheless, methods like Novel Class Discovery (NCD) and Generalized Category Discovery (GCD) have made strides by utilizing knowledge from labeled data to unearth new categories within unlabelled data in open world. However, our empirical studies reveal that supervised knowledge may not be universally beneficial, especially when there’s a significant semantic gap between the unlabeled and annotated datasets. To navigate this challenge, we propose a theoretically grounded metric, transfer flow, and a comprehensive benchmark with varying semantic similarities, aiming to precisely quantify the effectiveness of supervised knowledge under diverse conditions. Additionally, we have crafted two strategies that have demonstrably led to substantial improvements. Balance, aimed at securing an equitable sample distribution among all dataset categories, tackles the challenge of ensuring fair representation across classes, which is often skewed by the power law effect in real-world data. This imbalance or long-tailed nature of datasets often leads to model bias towards more frequently represented classes and poor generalization to less represented ones, thereby compromising the effectiveness of models in a variety of real-world applications. To combat the resulting long-tailed distribution, we initially proposed the imbalanced GCD method, which navigates the complex distribution of unlabeled real-world data. Subsequently, we developed a framework that integrates an Optimal Transport-based Expectation Maximization (Otem-IGCD) strategy with the objective of balanced class learning. This innovative method has proven its merit, marking a significant performance, thereby demonstrating its efficacy in overcoming the inherent challenges of data imbalance. Correctness, signifying the accuracy of dataset annotations, is critical for the accuracy and effectiveness of data-driven models. Achieving a fully corrected annotated dataset in the real world is challenging, requiring substantial investment and effort, with the end result still potentially marred by noisy labels. To address this, I introduce “Noisy NCD”, a practical setting that handles labeled data sampled from noisy datasets for NCD tasks. To address these challenges, I developed MKD-NCD, a strategy that utilizes mutual knowledge distillation to uncover new classes. Prior to this, we established SMKD, a mutual knowledge selection framework designed for closed-world environments. Together, SMKD and MKD-NCD have shown substantial success in navigating the complexities of noisy datasets. In this dissertation, I examine three principal data limitations in open-world scenarios: exhaustiveness, balance, and correctness. We address a fundamental problem associated with each limitation and provide effective solutions. Our research establishes a foundation for advancing Embodied Artificial Intelligence (EAI).