INQMAD: incremental streaming anomaly detection with density matrices, quantum measurement, and density estimation

Abstract Anomaly detection in data streams consists of detecting abnormal data samples in continuous data streams. This poses challenges that traditional and advanced anomaly detection methods are not equipped to deal with, such as the occurrence of conceptual drift and the need for continuous learning. Current state-of-the-art methods for detecting anomalies in data streams rely on fixed memory using hash functions or nearest neighbors. However, these methods can struggle to handle high-frequency data points or effectively identify seamless outliers, and also, they cannot be trained within a comprehensive deep-learning framework. In our research, we propose a novel method for streaming anomaly detection based on incremental density estimation. The method uses random Fourier features and incorporates the principles of quantum measurements and density matrices. We present the definition of the method and a variant of it, which can be represented as an exponential moving average density and a simple moving average, respectively. The method is designed to handle potentially infinite data streams and has a constant update complexity of O (1). To evaluate its effectiveness, we performed a comprehensive analysis by comparing the two variants with 12 state-of-the-art algorithms for detecting anomalies in data streams. We use 12 different streaming datasets to ensure an extensive evaluation of performance and efficiency.