Towards Explainable Artificial Intelligence: Insights from Selected Studies

In recent years, Artificial Intelligence (AI) has experienced increasing attention in academia, industry and politics. AI integrates concepts from diverse disciplines to create computer systems that can learn, reason, and solve problems much like humans do. Most recently, breakthroughs in Machine Learning (ML), neural networks, and generative models have greatly expanded the capabilities of AI. These advancements have led to significant improvements in AI systems, which are now applied across a wide range of fields. For example, AI helps doctors analyze medical scans for faster diagnoses, allows banks to spot fraudulent transactions, and lets manufacturers predict when factory machines need repairs to avoid breakdowns. In marketing, AI is used to create personalized ads by analyzing consumer behavior. It also enables chatbots that offer quick and direct customer support. Despite these achievements, there are clear examples where AI systems pose serious risks. Some facial recognition systems have shown racial bias by misidentifying individuals with darker skin tones more frequently than those with lighter tones. There have also been instances of chat systems producing inappropriate content, and self-driving cars facing safety risks when they fail to recognize obstacles. In many cases, AI systems lack transparency, a serious concern in areas like healthcare where understanding the reasoning behind a diagnosis is vital. Even advanced systems like ChatGPT, capable of generating human-like text, can spread hidden biases or misinformation because their inner workings are too complex to be fully understood. These examples underscore the potential hazards that can arise when the decision processes of AI systems remain unclear. In response to these challenges, the academic community has launched several research initiatives aimed at understanding and mitigating the risks associated with AI. Fields such as Trustworthy AI focus on developing AI systems that are reliable, transparent, and aligned with ethical principles, while research in Privacy-Preserving ML is dedicated to protecting sensitive data. Similarly, the field of Fair ML seeks to prevent biased behavior toward certain user groups. These efforts have not only advanced our technical understanding, but also highlighted the need for interdisciplinary approaches capable of managing the inherent complexities and risks of AI systems. Cross-disciplinary collaboration has led to the emergence of Explainable AI as a dedicated research field. Explainable AI addresses the black-box nature of AI systems by enhancing both their transparency and interpretability, enabling humans to understand their decisions. Research in this field investigates a broad range of topics, including the development of new methods to explain model behavior, the practical challenges involved with developing explainable real-world applications, and the integration of ethical considerations into system design. Current challenges within the field of Explainable AI are manifold, with one major issue being the disconnect between high-level theoretical methodologies and the development of practical frameworks. There are several technical hurdles that need to be overcome, including enhancing the inherent interpretability of ML models, evaluating the quality of model explanations, and ensuring their computational efficiency and scalability for use in real-time applications. In addition, recent research is exploring how the insights gained from model explanations can be used to improve model performance. Moreover, emerging AI regulations introduce new challenges and opportunities for Explainable AI. Given the broad scope of challenges, this dissertation aims to contribute to the field of Explainable AI by consolidating its current state of research and providing insights into selected issues. Therefore, it describes the results of five largely independent studies that address specific topics related to Explainable AI. The studies are discussed in more detail as follows. Study 1 consolidates the current state of Explainable AI research by conducting a systematic literature review from 2018 to 2024. Each year, a large number of studies related to Explainable AI are published. At the same time, various review studies are released, covering both broad and specific aspects of the field. In the first part of this study, we conduct a meta-review of 58 existing reviews to identify overarching streams, trends and underexplored areas of the research field. We observe that while Explainable AI has established itself as a key concept across a wide range of domains, there is limited understanding of the development and technical challenges involved with Explainable AI. In the second part of this study, we deepen our understanding of this issue by conducting a focused review on Explainable AI development. As explainability introduces unique requirements to the development process, we clarify the specific adaptations needed in the software development lifecycle, focusing on the steps and methods involved in designing and implementing Explainable AI systems. In total, 527 articles are analyzed, showing that while there is increasing efforts in implementing Explainable AI into practical applications, research in this area is still at an early stage. Study 2 addresses the challenge of data distribution shifts in ML, where model performance degrades as operational data diverges from training data. We investigate how model explanations can be utilized to enhance model robustness against such shifts. We introduce a novel hyperparameter tuning method, where the tuning process is informed by model explanations, with the goal of reducing overfitting and improving out-of-distribution performance. We evaluate the effectiveness of this method on a real-world energy consumption prediction dataset and validate our findings on two public benchmark datasets. The results demonstrate that our approach improves model robustness for both regression and classification tasks. In addition to robustness, it also contributes to explainability by clustering SHAP values, providing both global and local insights. Study 3 investigates how model explanations can be leveraged to boost the performance and generalization capabilities of semantic segmentation models. To examine this, we test an approach that integrates visual model explanations into the data annotation process. This leads to a more refined dataset, which in turn improves model performance through retraining. We evaluate this method on the public TTPLA dataset, widely used for detecting and segmenting power-grid hardware assets. The results show improved accuracy in semantic segmentation tasks as reflected by higher IoU scores across all evaluated categories. Moreover, incorporating model explanations enhances interpretability, providing insights into how specific features influenced the model’s predictions. Study 4 builds upon the findings of Study 3 by introducing textual explanations in addition to visual ones. This extension to multimodal explanations has only recently become feasible through large vision-language models, which combine visual understanding with language processing. While Study 3 relied solely on visual explanations, this study explores how textual explanations generated by GPT-4 Vision can provide clearer and more actionable insights. By combining visual evidence with textual reasoning, our approach offers contextual explanations (e.g., "The model confused a power line with a road marking due to visual similarity.") and practical recommendations (e.g., "Include more road-marking examples in training data."). We evaluate our approach on a diverse set of datasets spanning image classification, object detection, and semantic segmentation. The results demonstrate that GPT-4 Vision can generate textual explanations that offer interpretable justifications for model decisions, particularly in complex scenarios where visual explanations alone may be insufficient. Study 5 uses the context of three classification tasks to examine the feasibility of quantitatively evaluating AI systems against ethical principles outlined in the EU AI Act. We develop a framework that translates the principles of explainability, fairness, privacy, technical robustness and safety, as well as social and environmental well-being into measurable metrics. The framework encompasses a set of 22 different metrics and aggregates them into comprehensive scores to obtain both a dimension-specific and an overall compliance assessment. The experimental results demonstrate that a score-based approach allows for an initial evaluation of a model’s adherence to the AI Act. Applying our framework to the three unique datasets indicates its utility in revealing the relationships between the dimensions. Our experiments uncover noticeable differences in fairness, privacy and robustness scores across datasets, while explainability remains largely unaffected. This suggests that the different dataset characteristics such as size and distribution significantly affected these dimensions. However, these observations may be correlative rather than causative, requiring further research.