Implementing Artificial Intelligence in Safety-Critical Systems during Operation: Challenges and Extended Framework for a Quality Assurance Process

Implementing artificial intelligence (AI) in safety-critical systems comes with challenges that are also common in the implementation of AI in other domains. However, the consequences are distinct due to the inherent nature of safety-critical systems, where failure can lead to potential harm to people and the environment. This necessitates special consideration in AI design. When deploying AI in safety-related systems, it is necessary to ensure that the AI’s performance aligns with its training results and remains within acceptable bounds during operation to uphold the system’s safety. This paper focuses on addressing the quality assurance of AI during operation within safety-critical systems. Through a literature review and analysis, we aim to provide a comprehensive overview of common challenges encountered during operation and propose an extended framework of the ISO/IEC TR 5469 for a quality assurance process of AI within safety-critical systems. The extension adds monitoring of the AI component and a component for updating the AI, as well as several adjustments that are connected to those additions. As part of the monitoring, we introduce an indexing framework for the degradation of AI, based on a traffic-light design. The results, together with a comprehensive road map for future work, provide a foundation for further research and developing a holistic quality assurance of AI during operation.