Quantile Multi-Attribute Disparity (QMAD): An Adaptable Fairness Metric Framework for Dynamic Environments

Fairness is becoming indispensable for ethical machine learning (ML) applications. However, it remains a challenge to identify unfairness if there are changes in the distribution of underlying features among different groups and machine learning outputs. This paper proposes a novel fairness metric framework considering multiple attributes, including ML outputs and feature variations for bias detection. The framework comprises two principal components, comparison and aggregation functions, which collectively ensure fairness metrics with high adaptability across various contexts and scenarios. The comparison function evaluates individual variations in the attribute across different groups to generate a fairness score for an individual attribute, while the aggregation function combines individual fairness scores of single or multiple attributes for the overall fairness understanding. Both the comparison and aggregation functions can be customized based on the context for the optimized fairness evaluations. Three innovative comparison–aggregation function pairs are proposed to demonstrate the effectiveness and robustness of the novel framework. The novel framework underscores the importance of dynamic fairness, where ML systems are designed to adapt to changing societal norms and population demographics. The new metric can monitor bias as a dynamic fairness indicator for robustness in ML systems.