Explainable cross-modal face photo-sketch similarity with region-wise attribution and multi-extractor fusion

Abstract Cross-modal face similarity analysis remains challenging due to substantial modality discrepancies and the limited interpretability of existing recognition models. Although recent heterogeneous face recognition approaches have improved similarity estimation accuracy, they provide little insight into which facial regions guide similarity judgments or how these cues vary across modalities. To address this limitation, we propose an explainability-driven framework that integrates multi-extractor feature aggregation with region-wise attribution. We also present the Regional Consistency Similarity (RCS) formulation which summarizes localized attribution patterns derived from structured spatial perturbation and region-level aggregation. Multiple complementary feature extractors are jointly employed to reduce model-specific biases and provide stable similarity estimates under modality variations, while an optional natural-language explanation module converts region-wise patterns into user-friendly descriptions. Experiments on four photo-sketch datasets (CUFS, CUFSF, Toonify-CUFSF, and WildSketch) show that the proposed framework produces stable and spatially coherent region-wise attribution maps across diverse sketch styles and abstraction levels. Rather than pursuing recognition-accuracy gains, the multi-extractor configuration is interpretability-focused and recognition-preserving: it maintains verification performance comparable to the strongest single extractor while strengthening the region-wise consistency signal and exposing identity-relevant regions.