Quantifying the Stability–Recovery–Interpretability Trade-Off Between K-Means and Self-Organizing Maps for High-Dimensional Imbalanced Data

High-dimensional engineering datasets often combine class imbalance, noisy structures, and limited ground truth, making unsupervised analysis difficult to evaluate reliably. This study quantifies how three properties—partition stability, minority class recovery, and topological interpretability—are traded off across clustering methods, using a capacity-matched 25-seed comparison on a TCGA-derived RNA expression dataset (10,095 samples, 19 cancer types, 13,634 genes). We compare K-means across cluster counts k∈{19,…,400}, self-organizing maps (SOMs) across lattice sizes from 25 to 625 nodes, consensus K-means, a granularity-matched SOM-Super20 control, and four modern baselines (HDBSCAN, spectral clustering, Gaussian mixtures, and Leiden). At matched prototype budgets, K-means is both more reproducible and substantially better at recovering minority classes than SOMs: at 400 prototypes, K-means achieves pairwise NMI 0.819 versus 0.621 for the 20×20 SOM and recovers the smallest cancers 6–14× more effectively (pancreas effective coverage 0.760 vs. 0.054).Crucially, the SOM does not close this gap even when given more prototypes (0.07 at 625 nodes), so, under matched capacity, minority recovery is better explained by representational capacity and centroid allocation freedom than by topology preservation. The recovery is not free: increasing k overfragments the partition and lowers the pairwise ARI stability (0.643→0.419 from k=20 to k=400), while the NMI remains robust (≈0.82). The hardest minority, pancreas, is recovered only by high-capacity K-means and by no other method evaluated, including SOMs at any size, consensus K-means, SOM-Super20, HDBSCAN, Gaussian mixtures, spectral clustering, and Leiden. The SOM’s distinct value is therefore not stability or recovery but the interpretable two-dimensional topological visualization that it uniquely provides, including a gradient-organized structure that is reproducible across seeds for kidney (weaker for uterus). No single method optimizes all three properties; the appropriate choice depends on whether a task prioritizes reproducibility, minority recovery, or visual interpretability. Because these conclusions follow from the shape of the data and the allocation behavior of the algorithms rather than from biological semantics, we expect them to transfer to high-dimensional imbalanced engineering data, such as those from fault clustering, condition monitoring, and anomaly detection.

Authors

Institutions

Publication Details

Journal
AI for Engineering
Published
2026-08-20
DOI
https://doi.org/10.3390/aieng1020010
Primary Topic
Gene expression and cancer classification
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Quantifying the Stability–Recovery–Interpretability Trade-Off Between K-Means and Self-Organizing Maps for High-Dimensional Imbalanced Data

Imtiaz Ahmed, Hamdy Soliman
AI for Engineering
Gene expression and cancer classification
article

Quantifying the Stability–Recovery–Interpretability Trade-Off Between K-Means and Self-Organizing Maps for High-Dimensional Imbalanced Data

Imtiaz Ahmed, Hamdy Soliman
article en

Abstract

High-dimensional engineering datasets often combine class imbalance, noisy structures, and limited ground truth, making unsupervised analysis difficult to evaluate reliably. This study quantifies how three properties—partition stability, minority class recovery, and topological interpretability—are traded off across clustering methods, using a capacity-matched 25-seed comparison on a TCGA-derived RNA expression dataset (10,095 samples, 19 cancer types, 13,634 genes). We compare K-means across cluster counts k∈{19,…,400}, self-organizing maps (SOMs) across lattice sizes from 25 to 625 nodes, consensus K-means, a granularity-matched SOM-Super20 control, and four modern baselines (HDBSCAN, spectral clustering, Gaussian mixtures, and Leiden). At matched prototype budgets, K-means is both more reproducible and substantially better at recovering minority classes than SOMs: at 400 prototypes, K-means achieves pairwise NMI 0.819 versus 0.621 for the 20×20 SOM and recovers the smallest cancers 6–14× more effectively (pancreas effective coverage 0.760 vs. 0.054).Crucially, the SOM does not close this gap even when given more prototypes (0.07 at 625 nodes), so, under matched capacity, minority recovery is better explained by representational capacity and centroid allocation freedom than by topology preservation. The recovery is not free: increasing k overfragments the partition and lowers the pairwise ARI stability (0.643→0.419 from k=20 to k=400), while the NMI remains robust (≈0.82). The hardest minority, pancreas, is recovered only by high-capacity K-means and by no other method evaluated, including SOMs at any size, consensus K-means, SOM-Super20, HDBSCAN, Gaussian mixtures, spectral clustering, and Leiden. The SOM’s distinct value is therefore not stability or recovery but the interpretable two-dimensional topological visualization that it uniquely provides, including a gradient-organized structure that is reproducible across seeds for kidney (weaker for uterus). No single method optimizes all three properties; the appropriate choice depends on whether a task prioritizes reproducibility, minority recovery, or visual interpretability. Because these conclusions follow from the shape of the data and the allocation behavior of the algorithms rather than from biological semantics, we expect them to transfer to high-dimensional imbalanced engineering data, such as those from fault clustering, condition monitoring, and anomaly detection.

AI for EngineeringVol. 1(2)
New Mexico Institute of Mining and Technology (US)
Openalex Percentile: Top 16%
Gene expression and cancer classification
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.