Resource-Constrained Dynamic Kernel Density Estimation for Data Streams Anomaly Detection

With the rapid development of monitoring system and sensor technology, a large amount of data streams that continuously arrive with intensive data traffic have been generated in various fields. Anomaly detection is the core part of identifying the state changes of monitored objects. Considering that monitoring data streams is constantly generated in real time and not suitable for ex-post analysis, an anomaly detection method for data streams based on kernel density estimation is proposed to deal with the situation of limited computing resources. The method consists of dynamic kernel density estimation and dynamic threshold setting. In the dynamic kernel density estimation, a fusion mechanism for new data points based on the minimum distance constraint is established, and a merge strategy for buffer kernel functions based on the minimal spatial angle criterion is designed to realize the real-time probability density estimation of data streams under the limitations of storage and computation resources. In the dynamic threshold setting, the probability distribution fine-tuning strategy is adopted to update the initial thresholds online after the arrival of new data points. In view of the concept drift of data distribution, the increment and deletion strategy of the thresholds is set. In order to verify the validity of the method, simulation data, NAB standard data and industrial data were selected for analysis. The results show that the method can effectively and accurately perform online anomaly detection on large-scale data streams.