NADHS: Online Anomaly Detection for High-Dimensional Data Streams

Anomaly detection is a critical problem with a variety of applications since anomalies (which are unexpected observations that deviate from other observations significantly) pervasively exist. However, existing anomaly detection methods are mostly designed for batch processing and a certain type of anomaly. Accordingly, in this article, we propose a new framework, namely NADHS (<italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">oNline Anomaly Detection for High-dimensional data Streams</i>), for detecting various anomalies in high-dimensional data streams. In NADHS, we first propose a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Deep Subspace Sequential Clustering</i> (DSSC) algorithm where observations are grouped into clusters in real-time by exploring the DSC-Net (Deep Subspace Clustering Network) and SLC (Sequential Leader Clustering) algorithm. Afterward, we devise a <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">clustering-based anomaly detection</i> algorithm to recognize various types of anomalies. In addition, due to the dynamic variability of data and the difficulty in incremental updating of the DSC-Net, we devise a change point detection mechanism to capture the abrupt changes of data distribution. When an abrupt <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">change is detected</i>, we efficiently retrain the DSC-Net using the new arrival data and replace the online model with the new trained model to further improve the anomaly detection accuracy. Finally, we conduct extensive experiments over three real-world datasets to demonstrate the superiority of the proposed NADHS. The experimental results show that NADHS improves the anomaly detection accuracy by 37% at most compared to the state-of-the-art methods. The code is available at https://github.com/hey-jude-ajex/OD-NADHS.git.