Exploratory acoustic analysis of nonverbal affective and communicative vocalizations from non-speaking individuals with autism using cosine similarity

Nonverbal vocalizations from non-speaking individuals are challenging to interpret without context or familiarity. This study assessed acoustic similarities between 4727 affective and communicative vocalizations (e.g., frustration, delight, request) from 8 non-speaking individuals with autism using the ReCANVo dataset. We augmented the audio with noise, pitch shifts, and tempo changes, and extracted features using pre-trained Wav2Vec2 model representations, mel-spectrograms, and low-level descriptors to train a classification network (dim 24; 85/15 train/val). We calculated a mean vector (dim 24) for each class and computed their cosine similarities. Across eight individuals and seven vocalization classes, “Social” vocalizations were most globally similar (AvgCS = 0.950), suggesting acoustic features spanning affective and communicative states, while “Dysregulated” sounds were the most distinct (AvgCS = 0.886). However, the model (Acc = 0.660, MacroF1 = 0.555) was sensitive to speaker gender and age, and class imbalances, even after augmentation. Examining the participant with the most samples (P01; n = 1687; Acc = 0.676; MacroF1 = 0.695), “Request” was most similar to “Social” (CS = 0.978), suggesting shared underlying acoustics. “Delightedi” and “Frustrated” were the least similar (CS = 0.715), supporting the hypothesis that affective vocalizations are discernable via acoustical properties. Increased sample sizes and better real-world labeling techniques will enhance model accuracy, enabling interactive augmentative technologies for this population.