Agent-MIRA: AI-orchestrated medical imaging agent for PET image retrieval and assistance

Reporting on medical images can be time-consuming, especially in high-volume clinical settings. AI-agents designated to specific medical imaging tasks can potentially lead to improvements in clinical workflows. We present a prototype AI-agent to support clinical decision making by retrieving medical images which best match the patient images based on similarity and provide uncertainty estimation. The framework requires clinical metadata and PET images with lesion segmentation. A new patient's PET scan is processed by converting it to a feature vector representative of the image, which then enables the retrieval of the nearest feature vector neighbors by querying a database. Comparison between the radiomics and finetuned DINOv2 features was performed. Conditional uncertainty, an estimation based on feature significance, is calculated to state the level of confidence in similarity between patients. The AI agent, using DINOv2-derived features, retrieves a consistent set of patient cases that are phenotypically similar to the new patient. Each retrieved case is accompanied by clinical metadata, including cancer type, treatment history, and survival outcome. It also provides an estimate of the uncertainty in the matches and attention-based visualization to interpret the DINOv2 features. It is validated using eight independent patient test cases with benchmarking via clinical scoring to establish the level of support achieved for AI-agent orchestrated clinical decision-making. Scoring by the clinicians showed good correlation between the new patient and the retrieved database images with respect to the low uncertainty matches. We also integrated image-based retrieval with an entirely parallel text-embedding index of external clinical trials, thereby coupling case-based reasoning with evidence-based medicine in a single query interface using large language model.