Enhancing Chest X-Ray Classification with Retrieval-Augmented OpenAI GPT Models

In this study, we introduce an innovative method to enhance the classification of Chest X-Ray (CXR) images by integrating retrieval-augmented capabilities of OpenAI's GPT models. The proposed methodology involves encoding CXR reports using a pre-trained text encoder, analyzing X-ray images with GPT-4 to generate descriptive text, encoding these GPT-4 outputs, and then matching them to a database of encoded CXR report vectors for similarity. The most similar report vector is combined with the GPT-4 output vector and input into DenseNet201, a convolutional neural network, for disease classification. We evaluate the performance of our model using the CXR-14 dataset, which includes <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$\mathbf{1 0, 0 0 0}$</tex> X-ray images across five target pathologies: Pneumonia, Effusion, Cardiomegaly, Edema, and Emphysema. The dataset is carefully curated to ensure an even distribution of positive and negative cases for each condition. Our experimental results demonstrate that our model achieves comparable performance to established models such as GoogleNet and ResNet50 across the five thoracic pathologies, with AUC values ranging from 78.6% to 87.4%. This study highlights the potential of integrating retrievalaugmented GPT models with deep learning architectures for improved diagnostic accuracy in chest X-ray image classification.