Towards Safe AI: A Compositional Construction of Safe-visor Architecture for Interconnected Systems

This paper focuses on ensuring the safety of interconnected systems that utilize AI-based controllers, which have not been formally verified, for executing complex tasks. The approach involves encapsulating the AI-based controller within a Safe-visor architecture, which includes a Supervisor to monitor inputs from the unverified controller and a Safety Advisor to provide fallback mechanisms that maintain the system’s overall safety. For systems comprising multiple subsystems, the Safe-visor is constructed in a compositional manner. This involves designing the Safety Advisor and Supervisor for each subsystem individually and then integrating them using assume-guarantee reasoning. In developing Safe-visors for subsystems, the notion of hybrid controlled invariant sets is employed, with safety properties defined as linear temporal logic formulas over finite traces. The effectiveness of this approach is demonstrated through a case study involving a vehicular platoon.