Connecting Science and Stories: A Human-AI Method for Multi-Modal Knowledge Capture that Integrates Sensors, Community Narratives, and Computational Decision Models

Traditional decision support systems often fail to bridge scientific models with community knowledge, creating an implementation gap. The AI-enabled Modeling project (AIM) presents a novel approach that semantically links computational narratives with geospatial sensor data and physics-based models across domains. By leveraging large language models, semantic model integration, and multi-modal analysis, we establish technical services that enable integration of qualitative and quantitative knowledge systems. Our framework transforms unstructured stakeholder narratives into model configurations, generating culturally relevant representations of past events for complex socio-environmental challenges like hurricane impacts. The AIM project leverages, DataX, a data portal built on reusable science gateway components, incorporating CKAN-based data discovery, semantic variable mapping, interactive geospatially sensed visualization through Potree point cloud rendering, and a model integration platform. We demonstrate how unstructured interview transcripts can be systematically linked to data or physics-based models through a combination of natural language processing, standardized scientific vocabularies, and reusable computational cookbooks. This infrastructure enables transdisciplinary teams to connect lived experiences with observational data and computational simulations for technical integration in complex socio-environmental systems analyses.