Ensemble-based clustering and classification pipeline for cancer diagnosis using gene expression data

Accurate analysis of gene expression data is essential for understanding cancer mechanisms and improving diagnostics. However, the high dimensionality and heterogeneity of transcriptomic profiles often produce unstable clustering and classification outcomes. This study proposes an ensemble-based pipeline designed to improve robustness, interpretability, and clinical utility in cancer diagnostics. We developed a hybrid framework that integrates the Self-Organizing Tree Algorithm (SOTA) with agglomerative and spectral consensus clustering. Gene expression data from 6310 samples and 18,564 genes across 14 classes were transformed into cluster-based subsets. Classification was performed using Random Forest models with Bayesian hyperparameter optimization and out-of-fold stacking. Clustering quality was evaluated using the Relative Cluster Separation Index (RCSI), Calinski–Harabasz, Silhouette, and PBM indices, while biological interpretation was based on KEGG enrichment and Cytoscape (ClueGO/CluePedia) functional networks. The spectral consensus variant of SOTA achieved the most balanced performance across all metrics. On TCGA data, three- to five-cluster configurations provided high diagnostic accuracy (Accuracy ≈ 0.975, F1 ≈ 0.976) with statistically validated improvements ( p < 0 . 05 ) over baseline models. External validation on the independent expO dataset confirmed cross-platform robustness (Accuracy ≈ 0.811, F1 ≈ 0.843). Functional enrichment linked each cluster to biologically coherent pathways, including immune regulation, neuroactive signaling, metabolism, and viral response. The proposed ensemble–clustering–classification framework unifies consensus clustering, ensemble learning, and pathway-level validation, offering a reproducible and interpretable approach for cancer gene expression analysis, biomarker discovery, and precision oncology applications. • Hybrid pipeline integrates SOTA with consensus clustering. • Cluster-derived features enable perfect cancer classification. • Spectral consensus improves robustness and accuracy. • KEGG/ClueGO confirm biologically coherent gene clusters. • Framework supports biomarker discovery and stratification.