Confidence Based Sequential Drift Detection Under Resource Constraints

Concept Drift Detectors are the methods that are used to identify the patterns and behaviors of data that changes over time. These methods are tested separately on different datasets, which makes it difficult to compare which detector is suitable for particular situation. This paper presents the comparison of seven concept-drift detectors: ADWIN, DDM, EDDM, KSWIN, Page-Hinkley, Naive-T and a new lightweight detector called SeqConf. These seven detectors were tested on 120 simulated data streams. This experiment used two basic problems, three types of drift and twenty different random seeds for each case. Here we already know the drift points, the performance of each detector could be measured accurately. All detector used the same fixed online classifier, so that their detection performance, false alarms and computational requirements could be compared under the same conditions. SeqConf combines confidence-based monitoring, it does not require labels after deployment and uses constant memory. It uses persistence rule to reduce false alarms caused by single unusual samples. The results from 840 detector run shows that it has lowest memory usage and second lowest false alarm rate with 0.30 false alarms per 9000 samples on stationary data. Its detection rate is lowest among the seven detectors, at 12.5%. This experiment also showed that confidence-based detectors such as KSWIN, Page-Hinkley and SeqConf had difficulty detecting a gradual drift in the SEA dataset. EEDM, KSWIN and the Naïve-T baseline, produced alarms very frequently. The paper provides the complete experimental results and figures so that experiments can be repeated and verified.