Decision support for in-operation monitoring of the WEST tokamak first wall using multimodal large language model (LLM) on infrared imaging

First wall protection is a critical challenge in fusion machine operation, requiring rapid and precise diagnostic capabilities. The infrared diagnostics of long-pulse fusion machines, generate a large amount of video data. At WEST, the analysis of infrared thermal data is currently performed by Plasma-facing component Protection Officers (PPOs). This work presents a novel multimodal Large Language Model (LLM) approach to support PPOs in their diagnostic tasks. The framework addresses multiple challenges in fusion machine first wall monitoring by integrating infrared data (images and temperature curves) with domain-specific expertise. Key features include the preservation of data privacy, capturing a priori expert knowledge, quick post-pulse analysis capabilities, and the incorporation of expert feedback loops. The system employs prompt engineering and Low-Rank fine-tuning to embed domain knowledge directly into the diagnostic process. An ablation study using synthetic data demonstrated that a false recognition rate below 0.01 for hot spot characteristics could be achieved with 100 training samples. Based on these findings, the model was fine-tuned using 150 pairs of real-world images and corresponding prompts. This tool, once appropriately fine-tuned, was successfully implemented in the WEST control room during the C10 (10/2024 - 12/2024). Performance evaluation showed that of all the feedback collected, 32% were rated as ”OK” and 41% as ”positive response”. This development marks a significant advancement in automated diagnostic capabilities for fusion operations.