Towards Efficient Universal Audio Analysis: A Low-Complexity Model via Synergistic Multi-Task Learning

Audio analysis is a cornerstone of modern human-computer interaction, powering applications in smart devices such as phones, watches, and speakers. Key tasks enabling this interaction include voice activity detection, speech command recognition, and age and gender identification. While large-scale models can seamlessly integrate these tasks without significant performance degradation, their computational cost prohibits deployment on resource-constrained devices. In contrast, compact models suitable for embedded systems face a significant challenge: their limited parameter budget makes integrating multiple tasks without destructive interference difficult, and the optimal weight-sharing strategy is highly dependent on the specific task combination. This paper presents an analysis of weight-sharing architectures for multi-task learning on audio data. We systematically investigate the synergies and conflicts between tasks, evaluating pairwise and higher-order combinations. Based on our findings, we propose a novel low-complexity model that simultaneously executes four core tasks—voice activity detection, speech command recognition, age and gender identification—with fewer than 57,000 parameters. This represents a 53\% reduction in model size compared to a naive ensemble of single-task models. Our work not only achieves state-of-the-art performance on each individual task but also reveals that while most tasks exhibit positive synergy, the introduction of a more complex task, such as speaker diarization, can lead to performance degradation in larger task sets, highlighting the importance of careful task selection and architectural design for stable multi-task learning.