Drift Detection Methods on Machine Learning Systems: a Discussion over Discrete Live Data

Context: Machine learning has become an essential tool for addressing complex problems in information systems, encompassing industrial, commercial, and residential applications. Problem: Machine learning systems without frequent retraining are prone to data and concept drift, compromising predictive accuracy. This issue is particularly critical in scenarios where retraining is infeasible due to high computational costs or data unavailability. Solution: This study evaluates the performance of drift detection methods in discrete time series with controlled changes in mean and standard deviation using synthetic Gaussian signals. IS Theory: The General Systems Theory underpins the study by emphasizing how the interplay between drift detection and adaptive systems contributes to maintaining stability and efficiency in dynamic environments. Method: Experiments were conducted with variations in mean, standard deviation, and both parameters simultaneously in order to obtain qualitative patterns of drift detectors behaviors. The detectors ADWIN, KSWIN, and Page-Hinkley were tested under this scenario. Summary of Results: The findings reveal that ADWIN and Page-Hinkley exhibited greater precision and robustness, while KSWIN showed excessive sensitivity, leading to a high number of false positives. Contributions to the IS Field: This research offers a comprehensive analysis of drift detectors’ performance, specifically in scenarios involving changes in mean and standard deviation, providing useful reference for designing resilient machine learning-based forecasting systems. Impacts on the IS Field: The study advances the development of information systems that can adapt to dynamic data environments characterized by shifts in mean and standard deviation, with direct applications in industrial contexts and energy management.