Anomaly Detection in High-Dimensional Data Streams Using Machine Learning and Deep Learning Classifiers

Anomaly detection is a challenging task in highdimensional data streams. Applications like fraud detection, system monitoring, and Cybersecurity identifying the patterns are challenging because these applications contain an imbalanced nature and require robust models that maintain high accuracy while minimizing false positives. The traditional models like Isolation Forest and Local Outlier Factor (LOF) are not suitable to discover the hidden patterns. In this scenario, the advanced machine learning classifier provides the best result. Our objective in this paper is to use the advanced machine learning classifier to detect anomalies in high-dimensional dataset environments. To investigate the impact of hyperparameter adjustment on the F1-Score and overall anomaly detection accuracy. We used the 4 types of datasets (tabular, time-series, image-based, and text-based) to evaluate the model's performance. In this study, we used algorithms like Isolation Forest, One-Class SVM, LOF, and KNN, as well as advanced techniques like Autoencoders, Transformers, and Gradient Boosting. To determine the best algorithms for dealing with imbalanced datasets using criteria such as specificity, false positive rate, and recall. The experimental observation reveals that the algorithm Transformers identified as the best algorithm in comparison to other models, particularly for time-series and image-based datasets, achieving the highest F1-Score (95.4%) and PR AUC (94.1 %). We also used hyperparameter tuning to see the model's improvement. The observations reveal that OneClass SVM and Autoencoders benefit the most. Transformers demonstrated robustness by achieving the highest specificity (96.1 %) and the lowest false positive rate (3.9 %) on imbalanced datasets. Simpler models, like LOF and KNN, are computationally efficient.