Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System
Medical imaging underpins modern diagnosis, yet the interpretation of scans such as brain MRI, chest radiographs, dermoscopic images, and retinal fundus photographs still depends on scarce specialist expertise, and the computational tools that support it are usually built as isolated, single-modality applications. This paper presents a Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System that brings four imaging modalities into one explainable and knowledge-grounded platform. Uploaded images are standardised and analysed by a pretrained DenseNet121 backbone; Gradient-weighted Class Activation Mapping (Grad-CAM) converts the network’s internal activations into a spatial heatmap from which condition-wise scores and a leading condition are derived. The leading condition then drives a Retrieval-Augmented Generation (RAG) pipeline, built on a ChromaDB vector store with all-MiniLM-L6-v2 sentence embeddings, that returns curated findings, symptoms, and recommended clinical actions rather than freely generated text. Each analysis is logged in SQLite and can be exported as a structured PDF report through a Streamlit interface. The architecture is modular, traceable at every step, and deliberately positioned as decision support rather than a replacement for clinical judgement. We describe the design, implementation, and functional evaluation of the prototype, discuss its limitations, and outline the path toward clinically validated deployment.
Authors
- Yusuf Perwej (ORCID: https://orcid.org/0000-0001-8377-5829)
- Naman Mishra (ORCID: https://orcid.org/0000-0001-9374-6298)
- Abhishek Rai
Institutions
- Shri Ramswaroop Memorial University (IN)
Publication Details
- Journal
- International Journal of Scientific Research in Computer Science Engineering and Information Technology
- Published
- 2026-11-14
- DOI
- https://doi.org/10.32628/cseit261269
- Primary Topic
- Artificial Intelligence in Healthcare and Education
- Type
- article
- Field-Weighted Citation Impact
- 0.00