Multimodal GeoAI: An integrated spatio-temporal topic-sentiment model for the analysis of geo-social media posts for disaster management

The analysis of online communication on social networks has become a central research interest to improve disaster management. Especially geo-referenced textual posts have been investigated extensively using techniques from Geospatial Artificial Intelligence (GeoAI), topic modelling and sentiment analysis. However, workflows are traditionally sequential with independent processing steps, limiting their ability to capture interconnections between modalities and risking chains of dependencies. To overcome these limitations, we introduce an integrated GeoAI model called the Joint Spatio-Temporal Topic-Sentiment (JSTTS) model. Our proposed method combines semantic, sentiment, spatial and temporal knowledge into continuous feature vectors and is capable of computing geographically delineated sentiment-associated clusters of topics with meaningful location and temporal information. The properties of the JSTTS model were validated experimentally and the approach was evaluated against a comparable sequential workflow. Overall, our JSTTS model achieved higher topic quality scores with an average of 0.145 compared to 0.034 for the sequential workflow and higher average sentiment uniformity with an average of 0.89 versus 0.73. At the same time, both approaches exhibited similar spatial and temporal variance. As a secondary result, the Geographic Growing Self-Organising Map (Geo-GSOM) was developed to cluster multimodal feature vectors meaningfully in geographic space. It was evaluated on artificial training data where it reproduced up to 90% of the spatial autocorrelation while the non-spatial Growing Self-Organising Map (GSOM) only achieved 55%. The learned neuron grid can also be interpreted geographically. The utility of the JSTTS approach is demonstrated through a case study on the 2021 Ahr Valley flooding in Western Germany, where it identified interpretable multimodal clusters, a subset of which proved relevant for disaster management. The approach can be extended to other use cases and adapted for different modalities, holding potential for numerous follow-up studies.