Unsupervised Topic Shift Detection in Chats

In modern digital communication, chat platforms generate vast amounts of unstructured conversational data. However, the fluid, informal, and cross-timeframe nature of these chats challenges traditional supervised topic detection approaches. Addressing this gap, we propose an entirely unsupervised framework for detecting topic shifts in chat logs. Our method first converts individual messages into contextual embeddings using DistilBERT, thereby capturing nuanced semantic features. Next, rather than simply segmenting messages based on fixed temporal windows, we analyze the temporal evolution of these embeddings using a novel smoothing algorithm that highlights significant changes in the semantic trajectory of the conversation. Candidate topic shift points are then identified through an unsupervised peak detection process applied to the smoothed signal. Finally, an adaptive clustering algorithm groups the segmented data to refine topic boundaries without any manual labeling, ensuring the approach remains completely unsupervised. This method not only better accommodates the dynamic characteristics of chat logs but also robustly distinguishes fine-grained topic transitions. Experimental results demonstrate that our unsupervised framework outperforms traditional lexical and statistical techniques in detecting nuanced topic changes, making it highly effective for applications in social media analysis and real-time chat monitoring.