Optimized Gene and Image‐Based Feature Selection Using Modified Genetic Algorithms and Deep Learning for Predictive Skin Cancer Detection

ABSTRACT Gene selection is critical for cancer diagnosis because the ability to discover specific biomarkers has a major impact on diagnostic accuracy. Traditional approaches frequently struggle with high‐dimensional genomic data, where irrelevant or redundant characteristics might impair machine learning algorithms. Despite advances in computational approaches, there is a gap in the optimization of deep learning models for gene selection, particularly in terms of selecting the best model architecture and hyperparameters. This paper addresses three critical challenges in genomic biomarker discovery for cancer diagnosis: (1) the high‐dimensional nature of gene expression data, (2) the need for biologically interpretable feature selection, and (3) the optimization of deep learning architectures for genomic analysis. We present a novel hybrid approach combining modified genetic algorithms with deep neural networks to overcome limitations of traditional methods in handling feature redundancy and computational complexity. Our methodology introduces three key innovations: a dynamic mutation operator that adapts to population diversity, multi‐objective optimization balancing classification accuracy with biological pathway relevance, and simultaneous co‐evolution of both gene subsets and neural network architectures. The proposed system achieves state‐of‐the‐art performance, with 99.1% accuracy, 98.9% AUC‐ROC, and 99.0% F1‐score on the ISIC 2020 dataset, while maintaining clinically relevant sensitivity (98.0%) and specificity (98.5%). Extensive validation across six benchmark datasets demonstrates consistent improvements over existing machine learning and deep learning techniques, particularly in handling rare cancer subtypes and low‐resolution images. Future research directions include: (1) integration of multi‐modal clinical data to enhance rare subtype detection, (2) development of federated learning frameworks for privacy‐preserving distributed analysis, and (3) creation of explainability tools to bridge the gap between computational feature selection and clinical interpretation. The results establish our evolutionary optimization approach as both a high‐performance diagnostic tool and a flexible framework for advancing precision oncology research.