AI support for continuous open-ended situation awareness and scenario simulation: Design and early experiments in military intelligence

Combining large language models (LLMs) with structured domain knowledge representation (knowledge graph) and software agents appears to have potential for supporting domain-specific, open-ended situation awareness and scenario building. Inherent to these processes is that there is a complex and dynamic phenomenon under investigation, required input data is gathered from both structured and unstructured sources with varying confidence and quality levels, and process execution requires both human-human and human-AI collaboration. In this paper, we present a design, implementation, and partial validation for a data processing pipeline consisting of an LLM-based AI implementation and a collection of software agents, applied to a process of military intelligence. Our contributions are: 1) survey and eight-phase guidance of knowledge representation for LLMs, 2) survey and guidelines of trust in AI application adoption, and 3) implementation of agentic AI data processing pipeline for synthetic observation generation and processing. Our key findings indicate that through structured domain knowledge support, the foundation model is capable of generating outputs subject to domain expert evaluation, and that the utilization of software agents allows for experimenting with potential course-of-action (CoA) scenarios from a present situation. Future work requires more rigorous testing with domain experts to identify improvements needed in the data processing pipeline.