Chemical threat detection from open-source intelligence using a large language model: a zero-shot feasibility evaluation, with radiological assessment precluded by data sparsity

Abstract Timely detection of chemical and radiological incidents is crucial for public health surveillance; however, no global event-based surveillance system currently exists. Open-source intelligence (OSINT) and large language models (LLMs) present new opportunities for early detection; however, their application in these domains remains largely unexplored. This study evaluated GPT-4o’s performance for named entity recognition (NER) of chemical and radiological incidents from OSINT sources, assessing its potential to support global threat detection and early warning systems. Radiological entities were retained in the extraction schema throughout, but proved too sparse in the retrieved corpus to permit any evaluation of radiological performance; the evaluation reported here is therefore effectively confined to the chemical domain. A lexicon of 45 lay-accessible search terms, derived from 6646 technical chemical and radiological terms, was used to query the Bing News Application Programming Interface (API) from August 2024 to January 2025. Retrieved articles were scraped, translated when necessary, summarised by GPT-4o, and the summaries were and processed by GPT-4o using structured zero-shot prompts to extract entities, including event occurrences, substances, case numbers, fatalities, locations, and temporal features. Two annotators manually labelled a ground-truth dataset of 200 articles to evaluate performance using precision, recall, and F 1 Scores. Performance was compared against four baselines evaluated on the same corpus: a rule-based lexicon extractor, an off-the-shelf pretrained NER pipeline, a domain-pretrained biomedical NER model, and a supervised TF-IDF classifier. Of 37,618 retrieved articles, 8077 were parsed by GPT-4o. Event identification (F 1 = 0.96) and chemical substance recognition (F 1 = 0.94) were highly accurate. Geographic entities performed well at the state (F 1 = 0.98) and city (F 1 = 0.92) levels and remained strong at the country (F 1 = 0.93) and county (F 1 = 0.91) levels. Temporal (Date and Timeframe: F 1 = 0.74; Temporal expressions: F 1 = 0.81) and numerical extraction were moderate-to-strong (case numbers F 1 = 0.83; fatalities F 1 = 0.82). Only four positive radiological entity instances were identified within the annotated dataset, precluding meaningful evaluation of radiological extraction performance. Against a cross-validated supervised classifier, event identification improved from F 1 = 0.87 to F 1 = 0.96, and GPT-4o higher F 1 than the rule-based, pretrained NER, and domain-pretrained comparators across the entity types for which direct comparison was possible.Within the constraints of this limited evaluation, GPT-4o demonstrated feasibility for extracting several core public health entities from OSINT sources; however, while performance exceeded rule-based, pretrained-NER, and supervised classical baselines on a common corpus, comparison against transformer-based and fine-tuned biomedical NER architectures remains outstanding. Performance was weaker for Date and Timeframe precision. Radiological extraction could not be evaluated at all, because the corpus yielded too few positive radiological instances to support any performance estimate. This study, therefore, demonstrates chemical-domain feasibility under the conditions tested and documents radiological sparsity within this retrieval pipeline as a barrier to evaluation, rather than demonstrating radiological detection capability.