Enhancing Breast Cancer Diagnosis: Harnessing Deep Learning With Pretrained and Hybrid <scp>CNNs</scp> for Efficient Mammogram Analysis

ABSTRACT Deep learning has played a significant role in improving breast cancer detection using mammographic images. However, several limitations persist in existing studies, including imbalance in class distribution, inconsistent use of datasets, and dependence on individual convolutional neural network (CNN) models that may not generalize well across different data settings. To address these challenges, this study proposes a structured framework for breast cancer classification using pretrained CNN architectures and hybrid learning models. All experiments are conducted using the DDSM dataset to ensure consistency and avoid data‐related biases. A standardized preprocessing pipeline is applied to all images, incorporating intensity normalization, contrast enhancement, and noise reduction techniques to improve image quality. To further enhance model performance, data augmentation is applied selectively to the training set to mitigate class imbalance. Feature representations extracted from CNN models such as ResNet, DenseNet, VGG, and Inception are combined with machine learning classifiers including Random Forest and XGBoost. The experimental results indicate that hybrid approaches achieve more consistent and reliable performance compared to conventional CNN‐based classification. This study demonstrates that integrating deep feature extraction with ensemble learning methods can improve the robustness of mammogram classification while maintaining methodological rigor. The performance gains are influenced by the ROI‐based dataset characteristics and controlled experimental setup, and further validation on diverse datasets is required. The results are influenced by ROI‐based inputs and controlled experimental conditions.