Detection and classification of lung diseases in distributed environment

A significant increase in the size of the medical data, as well as the complexity of medical diagnosis, poses challenges to processing this data in a reasonable time. The use of big data is expected to have the upper hand in managing the large-scale datasets. This research presents the detection and prediction of lung diseases using big data and deep learning techniques. In this work, we train neural networks based on Faster R-CNN and RetinaNet with different backbones (ResNet, CheXNet, and Inception ResNet V2) for lung disease classification in a distributed and parallel processing environment. Moreover, we also experimented with three new network architectures on the medical image dataset: CTXNet, Big Transfer (BiT), and Swin Transformer, to evaluate their accuracy and training time in a distributed environment. We provide ten scenarios in two types of processing environments to compare and find the most promising scenarios that can be used for the detection of lung diseases on chest X-rays. The results show that the proposed method can accurately detect and classify lung lesions on chest X-rays with an accuracy of up to 96%. Additionally, we use Grad-CAM to highlight lung lesions, thus radiologists can clearly see the lesions’ location and size without much effort. The proposed method allows for reducing the costs of time, space, and computing resources. It will be of great significance to reduce workloads, increase the capacity of medical examinations, and improve health facilities.