Leveraging Spark-TensorFlow Distributor for Distributed Deep Convolutional Neural Networks: Accelerating large-scale COVID-19 Detection

The COVID-19 epidemic has been a critical global challenge due to its high mortality rate and rapid spread. Initial diagnostic methods, such as chest X-rays and reverse transcriptase polymerase chain reaction (RTPCR), can be time-consuming and require enhanced efficiency, especially in scenarios where large-scale testing is necessary. Combining deep learning (DL) and big data analytics tools is crucial for developing accurate and fast systems to detect COVID-19 infections and prevent further transmission. In this study, we introduce a distributed Deep Convolutional Neural Network (DCNN) framework designed for the detection of COVID-19, utilizing a comprehensive dataset of chest X-ray images. Our approach leverages the spark-tensorflow-distributor, which integrates TensorFlow 2.x with Apache Spark 3.x, enabling distributed training within the Spark framework. The system utilizes Spark’s barrier mode to execute distributed training tasks, significantly enhancing the speed of both training and testing phases. We classify chest X-ray images into two categories: COVID-19 and non-COVID-19. The experimental setup involves processing the large-scale dataset of chest X-ray images in a parallelized manner using the Spark framework. Our results demonstrate that the proposed DCNN model achieves a classification accuracy of 95.53%, with a testing time of 0.06 seconds and a training time of 18,909 seconds. Furthermore, when compared to other state-of-the-art methods for COVID-19 classification, our approach proves to be more efficient and effective. This study highlights the potential of combining distributed computing frameworks with deep learning techniques to address largescale medical image analysis challenges, particularly in the context of COVID-19 detection.