On Real-Time Monitoring of Machine Learning Systems

One of the main problems for discriminant machine learning models is the fact that the real data may differ from the data on which the model was trained and tested. Moreover, in many cases, it should be considered rather as a rule. The general set of data remains unknown, the real data can be arbitrarily different from the data used during the training phase. This means that generalizations of the model obtained at the training stage may not work, and the metrics obtained and confirmed by testing may not be performed. This, in turn, means that during the operation (industrial operation) of the machine learning model, it is necessary to monitor the correspondence between the characteristics of real data and the characteristics of training data. This is one of the main points in monitoring running machine learning models. The definition of the so-called data shift is necessary to determine the moment of decreasing the reliability of the results of the model, or to determine the fact that it cannot be further used without updating. At the same time, the data shift itself is heterogeneous in its impact and manifestation characteristics. The present article is devoted to the consideration of data shift analysis.