Delving into transfer learning within U-Net for refined retinal vessel segmentation: An extensive hyperparameter analysis

Blood vessel segmentation poses numerous challenges. Firstly, blood vessels often lack sufficient contrast against the background, impeding accurate differentiation. Additionally, the overlapping nature of blood vessels complicates separating individual vessels. Moreover, variations in the thickness of vessels and branching structures further augment complications to the segmentation process. These hurdles demand robust algorithms and techniques for effective blood vessel segmentation in medical imaging applications. The U-Net and its alternates have demonstrated exceptional performance related to conventional traditional Convolutional Neural Network (CNN). This study proposes a novel approach for retinal vessel segmentation through transfer learning. We proposed models such as VGG16 U-Net, VGG19 U-Net, ResNet50 U-Net, MobileNetV2 U-Net and DenseNet121 U-Net that employ pretrained models as encoders in U-Net architecture. We investigated the performance of pretrained models on DRIVE datasets with the optimizers Adam, Stochastic Gradient Descent (SGD) and RMSProp. The results revealed that models with Adam optimizer have shown better results. The evaluated results demonstrated that ResNet50 U-Net achieved the highest specificity of 0.9875, MobileNetV2 U-Net achieved a recall of 0.8056 and DenseNet121 U-Net attained an accuracy of 0.9689. VGG16 U-Net and MobileNetV2 U-Net have attained a dice coefficient of 0.849.