Affective features drive human representations of social touch expressions during naturalistic viewing

Touch is a potent communication tool. It has been suggested that a wide range of factors impact how we perceive social touch. No study has investigated which features we attend to when observing complex, naturalis-tic social touch expressions. To address this question, we curated 125 video clips showing a wide range of social interactions from the American TV series, Modern Family. Eighty participants watched 45 pseudo-randomly selected videos and performed a multiple arrangement task. This procedure produced the group-averaged pair-wise similarity judgments for social touch expressions. Visual, social, and affective features were extracted from each video clip using artificial neural networks (ANN), behavioural experiments, and human annotations. The combination of multiple regression and variance partitioning analyses revealed that selected affective, social, high-level visual, and ANN features collectively explained 52% of the variance in the perception of social touch expressions. Among these, affective features uniquely accounted for 33% of the variance. The current findings suggest that affective features, specifically whether the touch is used to convey positive or negative emotions, drive human perceptions of social touch during naturalis-tic viewing. Conversely, ANN features explained the least variance, suggesting that the models, trained on action perception and facial expression recognition, may not be sufficient to decode social touch expressions.