Analysing Chest X-Rays With Artificial Intelligence: Accuracy and Agreement With Radiologists

Introduction Artificial intelligence (AI) has changed our lives significantly in recent years and its integration into health system is often discussed. The ability to use AI as a tool to help radiologists detect X-ray pathologies is considered the next big step for radiology. AI could reduce doctor’s workloads and help radiologist to be more productive and accurately detect certain pulmonary pathologies such as nodules, mediastinal masses, pneumoperitoneum, fluid accumulation, and consolidation. [1,2] Aim To assess the real-world accuracy of AI in diagnosing various X-ray pulmonary pathologies. Methods A retrospective analysis was performed using the LUHS Clinic Electronic Patient System and medical image viewer MedDream. The study was approved by the Bioethics Center of Lithuanian University of Health Sciences (No.: 2024- BEC2-647). A total of 100 patients of various age groups who underwent a chest X-ray over the past year and had a diagnosis of pulmonary pathology based on ICD codes, were chosen. No specific patient selection criteria were applied. These X-rays were evaluated by a certified AI program as well as by two experienced radiology doctors. The AI program can detect nodules, mediastinal masses, pneumoperitoneum, fluid accumulation, consolidation. Diagnosis, formulated by an AI, and real diagnosis, formulated by radiologists, were compared. For data analysis, descriptive statistics, Kappa agreement coefficient and McNemar’s criteria were used. Significance level was set at p-value < 0.05. Statistical data analysis was performed using IBM SPSS Statistics 29.0. Results Results have shown that AI correctly identified the patient’s diagnosis in 59 cases, however, it was incorrect in 31 cases and partially correct in 10 cases, when AI provided an incomplete diagnosis. Thus, AI was fully correct in 59 % of all cases. AI has shown the best results in diagnosing consolidations (neoplastic or inflammatory) with a sensitivity of 100 %, however, AI struggled to diagnose pleural effusion with a sensitivity of only 67 %. The Kappa agreement coefficient was 0.526, with a p-value < 0,001. The Kappa value suggests that there is a moderate agreement between AI and radiologists’ evaluations. The sensitivity of the AI program in this particular study was 83 %, specificity 67 % and accuracy 74 %. What is more, the positive predictive value was 64 %, and the negative predictive value was 85 %. McNemar test was also used to evaluate paired differences between AI and physician evaluations. AI and physician evaluations differed statistically significant (p = 0.002). Conclusions To sum up, AI has a real potential to become a powerful tool to help doctors prevent mistakes and improve productivity. This study demonstrates that AI can assess various X-ray cases with moderate sensitivity and negative predictive value. However, it is still not capable of evaluating diagnoses with the same level of accuracy as experienced radiologists.