Optimization of Hadoop-Based Anomaly Detection Algorithm in Big Data Environment

In the big data environment, the distribution and nature of data change with time, and the traditional algorithm is not enough to carry out real-time anomaly detection. This paper introduced an anomaly detection algorithm based on Hadoop to improve the efficiency, accuracy and real-time of anomaly detection. First, the data is cleaned and features are extracted, and data is stored and managed using HDFS (Hadoop Distributed File System). Secondly, MapReduce framework is used to process large-scale data in parallel to improve computing efficiency. Then, the random forest algorithm based on Hadoop is introduced to realize anomaly detection. The performance of the algorithm is evaluated by accuracy, precision rate, recall rate and F1 value, and the algorithm is optimized by hyperparameter setting. The results show that the accuracy of the algorithm was improved from 0.87 to 0.92, and the resource utilization was up to 75%. The proposed method can effectively improve the accuracy of large-scale data anomaly detection. The research findings significantly enhance the accuracy and efficiency of large-scale data anomaly detection, providing a practical and effective solution to address the challenges of evolving data distributions and real-time detection needs.