False selection rate control in mixture models

Abstract The clustering task consists in partitioning elements of a sample into homogeneous groups. Most datasets contain individuals that are ambiguous and intrinsically difficult to attribute to one or another cluster. However, in practical applications, misclassifying individuals is potentially disastrous and should be avoided. To keep the misclassification rate small, one can decide to classify only a part of the sample. In the supervised setting, this approach is well known and referred to as classification with an abstention option. In this paper, the approach is revisited in an unsupervised mixture‐model framework. The purpose is to develop a method that guarantees the false selection rate (FSR) does not exceed a predefined level . We propose a plug‐in procedure and provide a theoretical analysis, quantifying the deviation of the FSR from the target with explicit remainder terms. Bootstrap versions of the procedure are shown to improve the performance in numerical experiments.