Multi-Fidelity Design Optimization (MFDO) for Fully Connected Deep Neural Network (FCDNN)

The Deep Neural Network (DNN), especially Convolutional Neural Networks (CNN), has revolutionized image pattern recognition by utilizing filters for feature extraction. Achieving superior classification results across diverse datasets relies heavily on identifying the optimal DNN structure and associated hyperparameter selection. However, determining this optimal configuration often requires significant computational resources and time. This paper presents a Multi-Fidelity Design Optimization (MFDO) approach, which quantifies deep learning structures and hyperparameters using the Genetic Algorithm (GA) optimization method. The MFDO efficiently explores the design space across various DNN hierarchical structures to attain optimal solutions. To reduce optimization time, the paper employs GA optimization on a smaller image dataset, followed by iterations involving training and cross-validation on a larger dataset. The DNN structure includes multiple modules with diverse choices such as convolutional layers, pooling layers, dropout layers, or fully connected layers, all optimized through encoded GA chromosomes. Training and validation are conducted using the MNIST and Fashion MNIST datasets. Results demonstrate the effectiveness of the GA process with the "low-fidelity dataset" in achieving shorter computational times, while validation with the "high-fidelity dataset" confirms the accuracy of the proposed design. The MFDO process effectively determines the optimal DNN structure with the best hyperparameters.