Beyond pixels: Zero-shot classification of chest X-ray reports via prompt engineering and LLMs

Accurately extracting structured clinical labels from unstructured chest radiology reports, particularly the MIMIC-CXR dataset, is critical for robust Clinical Decision Support Systems. However, conventional supervised deep learning faces major challenges: reliance on expert-annotated datasets, which are costly and time-consuming to curate, and the computational overhead of training or fine-tuning complex models. To circumvent these limitations, this study proposes a Prompt based Zero Shot Learning Framework that leverages the semantic reasoning capabilities of pre trained LLM to classify radiology reports without requiring gradient-based training or labeled data. We systematically investigate the impact of various text preprocessing pipelines and prompt engineering strategies, including task specific prompts, Persona based instructions, Chain of Thought reasoning, and Ensemble Prompting, alongside mechanisms for managing label uncertainty. Experimental results demonstrate that the optimal configuration, utilizing the IMPRESSION section of reports with specific uncertainty handling, achieves a weighted average F1 score of 0.85 and an Area Under the Curve of 0.92. These results significantly outperform traditional rule based labelers and unsupervised baselines in the domain of clinical text classification. This study substantiates the potential of NLP and prompt engineering to automate medical report structuring, offering a scalable and label free alternative to traditional supervised learning for healthcare applications. • Presents a novel zero-shot learning method for classifying chest X-ray reports using LLMs and prompt engineering. • Removes the need for labeled data and retraining, reducing annotation effort and computational cost. • Achieves superior performance compared to state-of-the-art models, with an average F1-score of 0.85 and an AUC of 0.92. • Shows that using only the IMPRESSION section yields the most accurate classification results. • Demonstrates strong potential for integration into clinical decision support systems.