Multi-scale and cardinality-based backbone: toward learning a general-purpose speaker representation

Abstract The extraction of speaker-related features through utterance embeddings has been extensively studied for years. Convolutional Neural Networks (CNNs), particularly deep Residual Networks (ResNets), have been widely used to capture spectral and hierarchical feature representations, enabling rich speaker embeddings. However, the development of robust and generalizable speaker representations remains a central challenge. Tailoring backbone architectures to the unique characteristics of speech is therefore crucial for effective embedding learning, raising a fundamental question: Which architectural principles yield the most robust and generalized speaker representations?. This study investigates CNN scaling dimensions as foundational design factors for speaker embedding backbones. The objective is to systematically investigate how different fundamental scaling dimensions influence Robustness and Generalization (R&G) in speaker embedding learning. We conduct a comprehensive evaluation by integrating these dimensions into two widely used ResNet-based baselines, ResNet-34 and ECAPA-TDNN. The experiments span both in-domain (Automatic Speaker Verification (ASV)) and cross-domain (Speech Emotion Recognition (SER)) tasks and are further supported by t-SNE and sharpness-aware generalization analysis through loss landscape visualization. The findings represent a step toward learning more robust and generalizable speaker representations through the integration of scale and cardinality dimensions. This design achieves smoother optimization trajectories, improved stability, and enhanced representational capacity over established backbones, demonstrating its broad applicability and effectiveness as a backbone network in speaker embeddings. Experimental results reveal consistent improvements in R&G across diverse conditions, highlighting the ability of multi-scale and multi-branch aggregated transformations to capture rich speech cues—such as pitch, tone, and phonation patterns—by jointly expanding the receptive field and feature diversity. This combination is valuable for capturing speaker traits that span both short- and long-term dependencies.