Enhancing Flood Prediction Reliability with RAFT-VLLM: A Retrieval Augmented and Fine-Tuned VLLM Approach for Geospatial Analysis

Vision Large Language Models (VLLMs) demonstrate significant potential for analysing complex geospatial and time-series data in food prediction; however, their reliability is frequently compromised by factual hallucinations, hindering practical utility in dynamic, real-time scenarios. This study introduces RAFT-VLLM, a hybrid framework designed to enhance the factual grounding and predictive reliability of VLLMs for short-term food risk assessment, specifically targeting the LLaVA-1.6 (7B) architecture. The framework uniquely combines targeted VLLM fine-tuning with a Retrieval Augmented Generation (RAG) pipeline. Fine-tuning is performed using a novel, domain-specific multimodal dataset meticulously engineered from ERA5 hourly reanalysis data, creating visual-textual pairs that link meteorological and hydrological states to food risk levels and justifications. Concurrently, the RAG system grounds the VLLM by dynamically injecting relevant historical meteorological data, pre-processed contextual information, and summarised current conditions—primarily derived from offline ERA5 archives—into the model’s prompt. This dual approach equips the VLLM with both specialised domain knowledge through fine-tuning and up-to-date, verifiable context through RAG. By pre-loading and injecting verified, up-to-date information, akin to emerging LLM-based interfaces in national weather services, this framework aims to significantly reduce hallucination rates, enhance the expert-level justification of predictions, and improve overall reliability in interpreting spatial weather plots for food risk forecasting. The study benchmarks this approach against VLLM baselines lacking these augmentations, demonstrating improved accuracy in food risk assessment.