Research on big data processing framework algorithm optimized by deep learning

This paper proposes a deep learning Optimization big data processing framework based on Hadoop distributed architecture, Convolutional Neural Network (CNN) and Whale Optimization Algorithm (WOA). First, the Hadoop distributed file system (HDFS) and MapReduce technologies are used to store and preprocess large-scale data in parallel, and reduce data redundancy and processing delay through efficient data distribution and cleaning. Secondly, as a feature extraction module, the CNN model extracts the spatial features of the data layer by layer by using multi-layer convolution and pooling operations to provide reliable feature representation for subsequent classification and prediction tasks. In order to further improve the performance of the model, this paper introduces whale optimization algorithm (WOA) to optimize the hyperparameters of the CNN model (such as learning rate, convolution kernel size, etc.). By simulating the hunting behavior of whales, WOA automatically adjusts the combination of hyperparameters during model training, thus significantly improving the accuracy and training efficiency of the model. The experimental results show that the proposed Hadoop-CNN-WOA framework has higher accuracy than the traditional single model in processing large-scale image classification tasks.