Self - adaptive canny edge detection with reinforcement learning and dominant texture color patterns for melanoma segmentation and classification in dermoscopic images

Abstract Melanoma, considered to be the most rapidly increasing cancer, has had great emphasis placed on its diagnosis. This paper proposes a powerful edge detection framework Self-adaptive Canny Edge Detection using Reinforcement Learning (CRL-Edge) that integrates Canny edge detection with reinforcement learning. This approach adaptively fine-tunes the threshold parameters of the canny so as to enhance the edge continuity particularly for images with weak boundaries. The research also focusses on proposing a feature extraction method Dominant Texture Color Patterns (DTCP) that effectively helps in classifying malignant melanoma from dermoscopic images. This method is proposed based on the extraction of texture and color features that are dominant in a particular local region. The RGB color channel that consists of texture patterns with more intensity variations is said to be a dominant texture feature and a color channel that has maximum color intensity variations is a dominant color feature. The texture-color patterns are combined together to form a pattern that is assigned a unique texture-color value that describes the image features. The proposed feature of texture and color is analyzed in dermoscopic color images to classify lesions as benign or malignant, using CatBoost, a gradient boosting technique. The CatBoost is compared with other gradient boosting algorithms like Random Forest, XGBoost and Light GBM. The experiments were conducted on two different databases, the ISIC Archive and the PH2 database. The images were evaluated, on the basis of performance metrics such as sensitivity, specificity, accuracy, precision, F1-score and AUC. The experiment results show that CRL-Edge segmentation provides better segmentation accuracy and the DTCP descriptor using CatBoost classifier provides enhanced classification accuracy for classifying malignant lesion. The new method is compared with different state-of-art methods and has demonstrated the best performance.