LLMs in Crisis Triage: Benchmarking Zero-Shot Classification of Social Media

Social media serves as a critical yet challenging data source during crises, requiring efficient triage of urgent information. To address this need, we evaluate 22 large language models on zero-shot classification tasks-identifying informative posts and classifying humanitarian categoriesfrom crisis-related social media datasets consolidated in CrisisBench. Leveraging the Lincoln Laboratory Super Computer, we benchmarked both open-access and commercial models, analyzing performance and reliability of responses across languages, datasets, and classes. Results reveal distinct model capabilities: smaller open-access models like Phi3-Medium:14b perform competitively in the simpler task of determining whether a post was informative; however for the more complicated task of classification into a number of humanitarian categories, larger models with more parameters, namely GPT-4o, performed the best.