CRB-NCE: An adaptable cohesion rule-based approach to number of clusters estimation

• A new cluster cohesion threshold-based rule system for number of clusters estimation. • Rules use tail ratios, robust and accurate indicators of deceleration of cohesion sequences. • Uses an optimization-driven approach to learn rules’ thresholds from synthetic data. • Thresholds can be adapted to specific clustering conditions by tailoring the learning data. • Outperforms or matches cluster validation indexes both in synthetic and real-world test data. Accurate number-of-clusters estimation (NCE) is a central task in many clustering applications, particularly for prototype-based k -centers methods like k -Means, which require the number of clusters k to be specified in advance. This paper presents CRB-NCE, a general cluster cohesion rule-based framework for NCE integrating three main innovations: (i) the introduction of tail ratios to reliably identify decelerations in sequences of cohesion measures, (ii) a threshold-based rule system supporting accurate NCE, and (iii) an optimization-driven approach to learn these thresholds from synthetic datasets with controlled clustering complexity. Two cohesion measures are considered: inertia (SSE) and a new, scale-invariant metric called the mean coverage index. CRB-NCE is mainly applied to derive general-purpose NCE methods, but, most importantly, it also provides an adaptable framework that enables producing specialized procedures with enhanced performance under specific conditions, such as particular clustering algorithms or overlapping cluster structures. Extensive evaluations on synthetic Gaussian datasets (both standard and high-dimensional), clustering benchmarks, and real-world datasets show that CRB-NCE methods consistently achieve robust and competitive NCE performance with efficient runtimes compared to a broad baseline of internal clustering validity indices and other NCE methods.