Med-Agent: A Hybrid AI Agent for Multimodal Cancer Diagnosis

Recent advances in artificial intelligence (AI) and multimodal learning have enabled new possibilities for holistic clinical decision support in the area of oncology. In this paper, we introduce MedAgent, an AI system that operates on three-dimensional dynamic contrast-enhanced MRI, structured clinicopathological data, and summarized narrative reports in clinical environments in order to permit holistic diagnostic reasoning and personalized therapeutic advice for breast cancer. MedAgent approximates the workflow of experienced clinicians via a sequence of tumoral segmentation, subtype prediction, and evaluation of tumoral aggressiveness via deep convolutional networks, followed by the combination of this information in a structured summary form fed into a large language model (LLM). The system includes a cross-modal fusion component tasked with aligning imaging and clinicopathological features via gated attention-inspired FiLM conditioning. Finally, the LLM component produces interpretable advice in line with established oncology protocols. Using the MAMA-MIA dataset (n = 1,506), performance evaluations showed MedAgent had high segmentation performance (Dice: 80.3%) as well as robust subtype prediction (AUC: 82.0%), with over 90% of its advice corresponding with either clinical protocols or specialist consensus. This investigation highlights the ability of multimodal AI systems to mirror multidisciplinary tumor board consultation, thus improving clinical interpretability as well as decision support at the individual patient level in scenarios of complex diagnostics.