Comparison of the Statistical and Autoencoder Approach for Anomaly Detection in Big Data

This paper compares two anomaly detection methods, comparing the Z-score statistical technique with autoen-coders for big datasets, which are crucial for industries like manufacturing, energy, and transportation to maintain smooth operations and avoid costly disruptions. Autoencoders outperformed Z-score statistical technique in anomaly detection on big datasets, achieving higher precision (0.94), F1-score (0.97), and recall (1.00) compared to Z-score statistical technique. This highlights autoencoders' superior ability to accurately identify anomalies, making them more effective for robust anomaly detection in complex data environments.