SUPERB-EP: Evaluating Encoder Pooling Techniques in Self-Supervised Learning Models for Speech Classification

Recent advancements in speech processing have been profoundly impacted by self-supervised learning (SSL) models, which effectively extract rich representations from multiple encoder layers. Notably, models like WavLM have shown out-standing performance in key tasks such as speaker identification (SID), emotion recognition (ER), and keyword spotting (KS). However, the optimal strategies for aggregating these representations remain under-explored. In this paper, we introduce SUPERB-EP, a benchmark framework designed to systematically evaluate various pooling techniques for speech classification tasks. We examine thirteen different pooling methods within the context of the WavLM large model across SID, ER, and KS tasks. Our results indicate that weighted average pooling yields the best performance for KS and SID tasks, while auto pooling excels in ER. These findings highlight the importance of carefully selecting pooling methods to enhance performance across different downstream tasks.