Assessment tool for trustworthy AI systems : operational workflows for compliance assessment with regulatory requirements

The CertAInty project has developed a comprehensive Certification Scheme for AI Systems to ensure their trustworthiness. The scheme defines clear objectives and corresponding means of compliance, criteria, and evaluation measures aligned with relevant standards and regulations. It offers technical and scientific methods for assessing key AI properties, such as transparency, human oversight, data governance, reliability, fairness, and safety and security. These methods support systematic evaluations throughout the AI lifecycle — from risk assessment, requirements and data acquisition to development, testing, and deployment. Based on an analysis of 38 standards (ISO/IEC, IEEE, EASA) and integration of EU legislation, the scheme bridges regulatory requirements with technical assessments. It identifies 95 techniques for measuring attributes like explainability, robustness, and security, ensuring compliance with the EU AI Act. To ensure a reliable and practical approach to AI certification, the CertAInty project uses Jira for requirements management. This tool ensures traceability, systematic documentation and real-time collaboration in line with standards such as ISO 9001, ISO/IEC 27001 and ISO/IEC 23894. Jira's features including version control, decision tracking and customisable dashboards improve efficiency and consistency throughout the certification process. Complementary MLOps practices ensure trustworthiness by design, incorporating best practices at every stage of the AI lifecycle. Our MLOps infrastructure supports model development, training, continuous integration and continuous deployment and monitoring, using tools such as GitHub, MLflow and Apache Airflow. This setup maintains traceability, automates testing and ensures ongoing standards compliance. The workflow has been validated in three high-risk use cases - medical applications, autonomous driving and vehicle detection at construction sites - aligning with the requirements of the EU AI Act and demonstrating effective certification processes for different data types and compliance requirements.