Contrastive Flip Probing for Bias Subspace Identification

Deep neural embeddings can encode latent factors that lead to unfair treatment or biased decision boundaries, even when explicit demographic or protected labels are unavailable in many practical scenarios. We propose Contrastive Flip Probing (CFP) to detect and mitigate these hidden biases in high-dimensional embedding spaces without relying on any ground-truth sensitive attribute. Our approach first infers a pseudo-attribute by unsupervised clustering or heuristic partitioning, then trains a surrogate classifier to differentiate the discovered subgroups. Next, we compute minimal-norm “flip vectors” that switch a sample’s predicted pseudo-label. By aggregating these flips across many instances and performing a low-rank factorization, we isolate a subspace that can be interpreted as encoding a potential bias factor. We prove theoretical guarantees under piecewise linear assumptions, showing that even approximate alignments between pseudo-labels and a true but unknown sensitive factor suffice to reveal a genuine bias sub-space. Removing or attenuating this subspace provides a pathway to reducing hidden biases in downstream tasks while preserving the majority of the original features.