Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System

Medical imaging underpins modern diagnosis, yet the interpretation of scans such as brain MRI, chest radiographs, dermoscopic images, and retinal fundus photographs still depends on scarce specialist expertise, and the computational tools that support it are usually built as isolated, single-modality applications. This paper presents a Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System that brings four imaging modalities into one explainable and knowledge-grounded platform. Uploaded images are standardised and analysed by a pretrained DenseNet121 backbone; Gradient-weighted Class Activation Mapping (Grad-CAM) converts the network’s internal activations into a spatial heatmap from which condition-wise scores and a leading condition are derived. The leading condition then drives a Retrieval-Augmented Generation (RAG) pipeline, built on a ChromaDB vector store with all-MiniLM-L6-v2 sentence embeddings, that returns curated findings, symptoms, and recommended clinical actions rather than freely generated text. Each analysis is logged in SQLite and can be exported as a structured PDF report through a Streamlit interface. The architecture is modular, traceable at every step, and deliberately positioned as decision support rather than a replacement for clinical judgement. We describe the design, implementation, and functional evaluation of the prototype, discuss its limitations, and outline the path toward clinically validated deployment.

Authors

Institutions

Publication Details

Journal
International Journal of Scientific Research in Computer Science Engineering and Information Technology
Published
2026-11-14
DOI
https://doi.org/10.32628/cseit261269
Primary Topic
Artificial Intelligence in Healthcare and Education
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System

Yusuf Perwej, Naman Mishra, Abhishek Rai
International Journal of Scientific Research in Computer Science Engineering and Information Technology
Artificial Intelligence in Healthcare and Education
article

Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System

Yusuf Perwej, Naman Mishra, Abhishek Rai
article en

Abstract

Medical imaging underpins modern diagnosis, yet the interpretation of scans such as brain MRI, chest radiographs, dermoscopic images, and retinal fundus photographs still depends on scarce specialist expertise, and the computational tools that support it are usually built as isolated, single-modality applications. This paper presents a Multi-Modal AI-Based Healthcare Diagnostic and Clinical Decision Support System that brings four imaging modalities into one explainable and knowledge-grounded platform. Uploaded images are standardised and analysed by a pretrained DenseNet121 backbone; Gradient-weighted Class Activation Mapping (Grad-CAM) converts the network’s internal activations into a spatial heatmap from which condition-wise scores and a leading condition are derived. The leading condition then drives a Retrieval-Augmented Generation (RAG) pipeline, built on a ChromaDB vector store with all-MiniLM-L6-v2 sentence embeddings, that returns curated findings, symptoms, and recommended clinical actions rather than freely generated text. Each analysis is logged in SQLite and can be exported as a structured PDF report through a Streamlit interface. The architecture is modular, traceable at every step, and deliberately positioned as decision support rather than a replacement for clinical judgement. We describe the design, implementation, and functional evaluation of the prototype, discuss its limitations, and outline the path toward clinically validated deployment.

International Journal of Scientific Research in Computer Science Engineering and Information TechnologyVol. 12(6)
Shri Ramswaroop Memorial University (IN)
Artificial Intelligence in Healthcare and Education
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon