The Black Diamond Metric: A Unified Framework for Detecting Systemic Bias in Machine Learning

In today’s rapidly evolving digital landscape, AI models have shifted from handling mundane tasks such as spam filtering to making high-stakes decisions in finance, law enforcement, and governance. However, these systems often perpetuate systemic biases. For example, The Markup’s 2021 investigation revealed that algorithmic lending toolsdenied mortgage applications from Black and Latino applicants up to 80% more frequently than equally qualified white applicants. Similarly,risk-assessment tools like COMPAS and PATTERN, widely used in bail and sentencingdecisions, systematically assign higher risk scores to Black defendants due to proxies suchas ZIP codes and arrest histories. To address this pervasive challenge, we introduce the Black Diamond Test (BD), amodel-agnostic metric that quantifies algorithmic bias as a single scalar. BD integrates: The sample-level mean absolute difference between predictions from a fair model(trained without sensitive attributes) and a full model (trained with all attributes) A χ2-based severity multiplier that captures group-level variance in biasUnlike existing fairness metrics, BD requires no sensitive attributes at inference timeand exhibits an interpretable “cliff” effect when disparity exceeds a critical threshold. We validate BD across five benchmark datasets (COMPAS, Adult, German Credit, BankMarketing, Heart Disease) and three model families (Logistic Regression, Random Forest,XGBoost), demonstrating its robustness and general applicability. The Black DiamondTest offers a unified, intuitive, and practical tool for auditing fairness in real-world AIsystems.