The predictability of the effectiveness of chains of classifiers in the out-of-domain detection

Models in machine learning (ML) can produce erroneous results when exposed to the out-of-domain (OOD) data. Thus the problem of correctly incorporating the detection of OOD samples should be considered for all production systems that use the ML component. The production system can incorporate the filtering of OOD in three ways: pre-filtering data, filtering it in-model, and posteriori filtering, with the option to chain them into larger ensembles. Considering the number of possible types of filters, as well the length of the ensemble, it is important to be able to predict the upper and the lower bound of the performance of such ensembles on the basis of the performance of individual filters. The paper analyses challenges associated with the construction of the model, and then proposes the model where the performance of any chain of filters can be predicted, without actually assembling and testing it. An experiment is used to demonstrate the feasibility of the model, using the simple digit classification task, and F1 as the metric, resulting in the prediction error well below 0.1.