A multi-cycle recursive clustering algorithm for the analysis of social media data streams

Abstract Events are usually embedded in latent topics and the extraction of these latent topics are enabled by event detection algorithms. Unsupervised algorithms like Clustering algorithms are very useful for detecting events but with requirements which may not be relevant or easy to determine when using unstructured textual social media data. For instance, some algorithms are required to be used on specific data shapes, but determining the shape of an unstructured data may not be practical aside from the high level of noise in the data. Many of the existing algorithms work well with structured data, however, some of these algorithms can be adapted to unstructured data with the caveat that cluster formations may not contain consistent contextual information. We propose a novel Multi-Cycle Recursive Clustering Algorithm (MCRCA), able to sequentially eliminate noise, resulting in high homogeneous cluster formations. MCRCA does not require the initial specification of clusters numbers as the estimated number of clusters can be deduced at convergence. Our algorithm out-performs the classical LDA and K-Means algorithms in forming highly homogeneous clusters, context-wise.