Exploring layer-wise features of whisper and wav2vec 2.0 models and their aggregation for infant cry classification in low-resource conditions

Infants use cry as their primary mode of communication, which they use to signal hunger, discomfort, illness etc., to their caregivers. Understanding and interpreting these cries can provide critical insights into their well-being and aid in timely intervention. However, manual analysis of infant cry is subjective and prone to variability, especially when parents/caregivers have limited experience. While deep learning (DL) based infant cry classification (ICC) systems can be handy in such situations, they need large datasets to train robust models, without which they become vulnerable to challenges like variability in cry patterns across infants, noise in real-world recordings, etc. One way to improve the performance in such low-resource conditions is to use transfer-learning, where features extracted from pre-trained model are used to build models for the task-at-hand. Motivated by this, we propose to build ICC system using features from pre-trained wav2vec 2.0 and whisper models and compare their performance with the traditional ICC system that uses MFCC features. Since different layers of wav2vec 2.0 and whisper might encode the required information differently, we first empirically determine their relevance to ICC. In order to best utilize the complementary information that may be present in the features extracted from different layers and time-steps, we then propose to fuse them together. We propose to use different fusion strategies such as statistical pooling, attentive statistical pooling, linear-merge and attention-merge based approaches to aggregate the features. Furthermore, we explore different deep-learning architectures such as LSTM and x-vector-based architecture to perform cry classification. Obtained results indicate that attention-merge on wav2vec 2.0 large features combined with x-vector gives the best performance.