From Symbolic Reasoning to Object Embeddings: Advanced Approaches of Knowledge Distillation in Compacted Neural Networks

The rapid advancement of neural networks has profoundly impacted industries worldwide. However, challenges arise when these networks are deployed on devices with limited resources. The two major impediments are the computational power required and the storage demands. To address these challenges, domain knowledge has emerged as a gateway to reducing the gap between architecturally distinctive networks. Among the various domain knowledge integration mechanisms, this dissertation explores Knowledge Distillation (KD), a process where knowledge extracted from a high-performance network is incorporated into a compact neural network. This approach enables the compact neural network to enhance its generalization capabilities while performing tasks that would otherwise be unattainable due to its limited architecture. The dissertation also examines the integration of logical reasoning which addresses the shortcomings of current data-driven machine learning approaches. Such data-driven learning approaches are regarded as ``black-box" which often lack adequate reasoning behind their predictions. By combining logical reasoning with knowledge distillation, this dissertation demonstrates significant performance enhancements for resource-constrained neural networks. This dissertation begins by providing a foundational approach to incorporating domain knowledge into machine learning, offering a roadmap that enhances traditional learning mechanisms. The core of this dissertation is the development and refinement of knowledge generation strategies. It starts with the creation of knowledge through weighted ensembles of logits derived from multiple high-performing models leading to accuracy improvements in predicting images in a multi-class image classification task. Additionally, the use of weighted ensemble of intermediate models further boosts the capabilities of compact models. This dissertation also introduces a novel neurosymbolic approach to knowledge distillation, which combines symbolic reasoning with sub-symbolic neural processing to establish a robust learning system. Diverging from earlier approaches, the latter part of this research explores the use of Concept Activation Vectors (CAVs) to enhance both interpretability and performance. These CAVs, derived from a high-performing model, are integrated into compact networks using a neurosymbolic approach. This integration creates a sophisticated framework that not only improves the interpretability of the distilled knowledge but also extends its applicability to image classification tasks. The class-specific features, converted into concept activation vectors improve performance in compact models, serving as an effective form of knowledge transfer. The final part of the research expands on the idea of knowledge distillation with object embeddings within a multi-modal architecture. Differing from the previous approaches, this work applies knowledge distillation for visual question-answering (VQA) tasks. Instead of relying on learning from raw pixels, this research focuses on generating rich knowledge of the images through object embeddings. These embeddings are composed of specialized features derived from regions of interest, structural attributes, and spatial relationships. The incorporation of such rich knowledge ensures enhanced performance on small networks. This dissertation offers contributions by introducing advanced techniques in knowledge distillation that optimize the performance and interpretability of compact neural networks. By focusing on integrating domain knowledge and incorporating logical reasoning, the research demonstrates how compact models can achieve enhanced functionality and greater generalization capabilities. The use of Concept Activation Vectors (CAVs) and object embeddings highlights the potential for these models to perform effectively in different applications, such as image classification and Visual Question Answering (VQA). The implication of this provides a pathway toward more efficient and interpretable AI systems that can be successfully deployed in resource-constrained environments.