Social Governance Oriented Multimodal Situation Perception and Bilateral Collaborative Scheduling Simulation

Social governance scenarios often involve multiple modalities of information. Effectively utilizing multimodal information to understand the situational context is an important issue. Based on the situation perception, conducting situational development simulations is also beneficial for selecting appropriate response strategies. This paper takes the handling of public petitions in social governance scenarios as an example, integrating multimodal information such as event descriptions, locations, times, and images for event clustering. Based on the event clustering, we construct LLM prompts and perform situational understanding through multi-layer perceptrons. Additionally, we propose using LLM-based dual-agent method for Bilateral Collaborative Scheduling Simulation, with the first LLM representing the petitioners and the second LLM representing the civil servants. The LLM which plays the role of civil servants cooperate with various departments according to an operation set, on the goal to satisfy the petitioners. Through the interaction between entities, we are able to evaluate potential optimal strategy. This paper constructs a real-world dataset for public petitions and validates the proposed method. We verify the effectiveness of the method through both quantitative and qualitative analyses.