LLaVA-MultiMammo: adapting vision-language models for explainable and comprehensive multiview mammogram analysis in breast cancer assessment

Mammogram analysis plays a crucial role in breast cancer screening and diagnosis, but traditional computer-aided detection/diagnosis (CAD) systems are constrained by single-task specialization, inadequate multimodal data integration, and limited explainability. This study introduces LLaVA-MultiMammo, a novel approach that adapts vision-language models for comprehensive and explainable multi-view mammogram analysis. By leveraging the capabilities of large language models, LLaVA-MultiMammo integrates multi-view imaging data with clinical text information to perform multiple tasks within a single framework. We trained and evaluated the model's performance on three critical tasks for mammogram interpretation: BI-RADS density categorization, malignancy classification, and 5-year breast cancer risk prediction. Compared to task-specific models, LLaVA-MultiMammo outperformed in density categorization (80.6% vs. 78.1% accuracy), showed similar performance in malignancy classification (AUC 0.755 vs. 0.742), and achieved comparable results in risk prediction (AUC 0.673 vs. 0.702). The model's structured conversational approach provides step-by-step reasoning that addresses the explainability gap in traditional CAD systems, potentially enhancing radiologistAI collaboration and leading to more transparent and comprehensive breast cancer evaluation in clinical practice.