The Fairness Threshold Problem: A Governance Framework for Resolving Metric Conflicts in Enterprise AI Fairness Assessment

Enterprise AI fairness assessment frameworks require organisations to evaluate AI system outputs against defined fairness metrics. The three primary metrics — demographic parity, equalised odds, and counterfactual fairness — are well established in the technical literature and form the foundation of regulatory fairness expectations under the EU AI Act, Australian anti-discrimination law, and sector-specific guidance in financial services and healthcare. What the literature and regulatory frameworks do not resolve is what an organisation must do when a system satisfies one fairness metric but not another — a condition that the mathematical literature has demonstrated is the rule rather than the exception in real-world AI deployments. This paper identifies the conditions of the Fairness Threshold Problem and argues that it is not a technical problem but a governance design problem. The Fairness Threshold Problem arises from three structural properties of real-world fairness assessment. First, metric incompatibility: the three primary fairness metrics cannot all be simultaneously satisfied in most real-world settings where base rates differ across protected groups, a result established by Chouldechova (2017) and Kleinberg et al. (2016). Second, threshold indeterminacy: different regulatory regimes, protected characteristics, and system risk tiers require different threshold levels, and no universal threshold table is consistent with all applicable obligations. Third, conflict indeterminacy: when a system satisfies one metric but not another, no technical criterion determines which metric takes precedence. The resolution is inherently a value judgement that a governance framework must make explicit and accountable. This paper introduces the Fairness Threshold Governance Framework: three interlocking governance obligations that together produce a structured, auditable, and defensible resolution of the Fairness Threshold Problem. The Primary Metric Selection obligation requires the governing body to designate, prior to deployment, which fairness metric takes precedence for each system and each protected characteristic group, and to document the governance rationale for that selection. The Threshold Calibration obligation requires that thresholds be set by system risk tier and protected characteristic, with defined escalation obligations when thresholds are breached on any metric. The Conflict Resolution Protocol specifies the governance process for resolving cases in which primary metric compliance is achieved, but secondary metric breach occurs, establishing that this is a Board-level governance decision with defined documentation, escalation, and disclosure implications. The paper analyses five major governance frameworks and regulatory regimes, demonstrating that none provides a structured resolution to metric conflict, nor does any impose a governance obligation regarding the selection of primary metrics.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-06-17
DOI
https://doi.org/10.5281/zenodo.20728390
Primary Topic
Ethics and Social Impacts of AI
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

The Fairness Threshold Problem: A Governance Framework for Resolving Metric Conflicts in Enterprise AI Fairness Assessment

M Maruf Hossain
Zenodo (CERN European Organization for Nuclear Research)
Ethics and Social Impacts of AI
preprint

The Fairness Threshold Problem: A Governance Framework for Resolving Metric Conflicts in Enterprise AI Fairness Assessment

M Maruf Hossain
preprint en

Abstract

Enterprise AI fairness assessment frameworks require organisations to evaluate AI system outputs against defined fairness metrics. The three primary metrics — demographic parity, equalised odds, and counterfactual fairness — are well established in the technical literature and form the foundation of regulatory fairness expectations under the EU AI Act, Australian anti-discrimination law, and sector-specific guidance in financial services and healthcare. What the literature and regulatory frameworks do not resolve is what an organisation must do when a system satisfies one fairness metric but not another — a condition that the mathematical literature has demonstrated is the rule rather than the exception in real-world AI deployments. This paper identifies the conditions of the Fairness Threshold Problem and argues that it is not a technical problem but a governance design problem. The Fairness Threshold Problem arises from three structural properties of real-world fairness assessment. First, metric incompatibility: the three primary fairness metrics cannot all be simultaneously satisfied in most real-world settings where base rates differ across protected groups, a result established by Chouldechova (2017) and Kleinberg et al. (2016). Second, threshold indeterminacy: different regulatory regimes, protected characteristics, and system risk tiers require different threshold levels, and no universal threshold table is consistent with all applicable obligations. Third, conflict indeterminacy: when a system satisfies one metric but not another, no technical criterion determines which metric takes precedence. The resolution is inherently a value judgement that a governance framework must make explicit and accountable. This paper introduces the Fairness Threshold Governance Framework: three interlocking governance obligations that together produce a structured, auditable, and defensible resolution of the Fairness Threshold Problem. The Primary Metric Selection obligation requires the governing body to designate, prior to deployment, which fairness metric takes precedence for each system and each protected characteristic group, and to document the governance rationale for that selection. The Threshold Calibration obligation requires that thresholds be set by system risk tier and protected characteristic, with defined escalation obligations when thresholds are breached on any metric. The Conflict Resolution Protocol specifies the governance process for resolving cases in which primary metric compliance is achieved, but secondary metric breach occurs, establishing that this is a Board-level governance decision with defined documentation, escalation, and disclosure implications. The paper analyses five major governance frameworks and regulatory regimes, demonstrating that none provides a structured resolution to metric conflict, nor does any impose a governance obligation regarding the selection of primary metrics.

Zenodo (CERN European Organization for Nuclear Research)
RED Consulting (Norway) (NO)
Peace, Justice and strong institutions, Reduced inequalities
Ethics and Social Impacts of AI
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.