Towards Quantifying Compliance with the EU AI Act

As AI systems proliferate in high-risk domains, assessing their compliance with emerging regulatory standards has become imperative. The EU AI Act outlines ethical requirements across five dimensions: explainability, fairness, privacy, robustness, and social and environmental well-being. However, existing evaluation approaches lack a unified methodology to quantitatively operationalize these principles. In this paper, we propose a structured, score-based framework that translates the Act’s pillars into 22 interpretable metrics, enabling reproducible, model-agnostic compliance assessments. Applied to three benchmark tabular classification tasks using a standardized deep learning model, our framework captures how dataset characteristics shape ethical performance. The results reveal key trade-offs: models with high predictive accuracy do not necessarily meet compliance expectations, and larger datasets tend to improve robustness but increase vulnerability to privacy leakage. Correlation analyses expose metric redundancy in fairness and explainability, suggesting potential for simplification. Privacy metrics, by contrast, remain essential and diverse. Social and environmental measures emerge as least mature, underscoring the need for novel, bounded metrics in future research.