Multi-scale autoencoder-based unsupervised framework for social behavioral structure discovery and anomaly detection

Understanding social behavior is crucial in neuroscience and psychiatry, as it provides insights into the mechanisms underlying social interaction and related disorders. Traditional behavioral analyses often rely on manually defined features, such as nose-to-nose contact, which are subjective and limited to pre-specified behaviors. In this study, we propose a multi-scale LSTM-based Autoencoder framework for unsupervised evaluation of abnormal social behavior in mice. The model learns behavioral representations from normal control mice by reconstructing pose trajectories extracted from long-term videos and computes anomaly scores when tested on groups subjected to different rearing conditions. To capture both rapid and prolonged behavioral dynamics, the proposed method integrates multiple temporal scales. Experimental results on the public CalMS21 dataset demonstrate that our model can effectively identify different social patterns in hidden layer. The proposed unsupervised approach provides a promising tool for quantitative assessment of animal behavior and opens new possibilities for discovering subtle behavioral phenotypes relevant to neurobiological research.