A modified hybrid CNN-based feature selection and ANN classification framework for patient-level breast cancer histopathology analysis

Histopathological image analysis plays a central role in breast cancer diagnosis; however, developing computational models that are both reliable and generalizable remains challenging due to inter-patient variability, dataset heterogeneity, and evaluation constraints. These challenges highlight the importance of validation strategies that emphasize reproducibility and robustness alongside predictive performance. In this study, we present a hybrid framework for breast cancer histopathology classification that combines transfer learning–based feature extraction with a lightweight artificial neural network (ANN) classifier. A pre-trained DenseNet121 model is employed as a frozen feature extractor to obtain high-level representations from histopathological images, followed by Random Forest–based feature selection to reduce dimensionality and improve feature robustness. The selected features are classified using an ANN trained with multiple optimization strategies, including a novel weight updating scheme, under identical experimental conditions. To ensure reliable evaluation, strict patient-level five-fold cross-validation is implemented using group-based data splitting, and predictions are aggregated at the patient level to mitigate information leakage. The framework is evaluated on two independent public datasets, BreakHis and BACH, to assess external generalization. Experimental results indicate that the evaluated optimization methods yield comparable classification performance across folds and datasets, with variations primarily influenced by feature representation and validation strategy rather than optimizer choice alone. Overall, this work emphasizes the role of careful validation design, patient-level evaluation, and external dataset testing in developing reliable breast cancer histopathology classification systems. The proposed framework provides a reproducible and computationally efficient baseline that can support future research toward clinically realistic and scalable diagnostic applications.