Challenges in Medical Image Processing and Analysis: Cross-Cutting Issues Between Imaging Modalities and Deep Learning Models

In recent years, with advances in computational power and the integration of Artificial Intelligence (AI) into healthcare systems, medical image processing (MIP) has seen significant benefits through the application of deep learning (DL) techniques. This application has made complex tasks, such as segmentation, classification, and reconstruction, more feasible, explainable, and automated across clinically diverse imaging modalities. However, with these benefits also come challenges in uniquely aligning DL solutions with the consistent constraints of these medical imaging modalities. To provide a comprehensive insight into these challenges, this study critically examines deep learning challenges across four key medical imaging modalities: MRI, CT, ultrasound, and histopathology. Their diversity in acquisition, resolution and annotation techniques makes them suitable for this consideration. X-ray imaging is excluded from this study due to its lower spatial complexity and standardised acquisition pipeline. Furthermore, this study examines modality-specific challenges in standard DL-based solutions across Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), transformers, and hybrid systems, and highlights their cross-cutting issues and intersection with the challenges seen in these images. Previous reviews often overlook the significant interactions between DL-based solution design and medical imaging characteristics. By highlighting these challenges, this study helps guide the design of stronger, more practical DL-based solutions that can make medical image processing more reliable and useful in real healthcare settings.