Identification and summarisation of events from Twitter using clustering algorithms and deep neural network

Abstract The proliferation of social networks has caused an increase in the amount of textual content generated by users. The voluminous nature of such content poses a challenge to users, necessitating the development of technological solutions for automatic summarisation. This paper presents a two-stage framework for generating abstractive summaries from a collection of Twitter texts. In the first stage of the framework, event detection is carried out through clustering, followed by event summarisation in the second stage. Our approach involves generating contextualised vector representations of tweets and applying various clustering techniques to the vectors. The quality of the resulting clusters is evaluated, and the best clusters are selected for the summarisation task based on this evaluation. In contrast to previous studies, we experimented with various clustering techniques as a preprocessing step to obtain better event representations. For the summarisation task, we utilised pre-trained models of three state-of-the-art deep neural network architectures and evaluated their performance on abstractive summarisation of the event clusters. Summaries are generated from clusters that contain (a) unranked tweets , (b) all ranked tweets , and (c) the top 10 ranked tweets . Of these three sets of clusters, we obtained the best ROUGE scores from the top 10 ranked tweets . From the summaries generated from the clusters containing the top ten tweets, we obtained ROUGE-1 F score of 48% , ROUGE-2 F score of 37% , ROUGE-L F score of 44% , and ROUGE-SU F score of 33% which suggests that if relevant tweets are at the top of a cluster, and then better summaries are generated.