Optimizing memory-efficient multimodal networks for image classification using differential evolution

This paper presents a novel approach to developing tuned model architectures and hyperparameter sets for multimodal classification, leveraging Differential Evolution to optimize for memory-efficient design (MEMODE). The evolved model combines image channel data with vectorized word matrices generated through trained Word2Vec embeddings, as a resourceful convolutional processing of text and image inputs. Additionally, to enrich the semantic representations drawn from text-based inputs, a pretrained ALBERT module is included to process a parallel input stream. The custom Differential Evolution algorithm employed utilises a penalized objective function that directly accounts for model size and Mean F1 performance. We focus on the critical task of hyperparameter optimization and neural architecture search, while accounting for a set design limit of 100 MB. Significantly, the network evolved through the MEMODE algorithm achieved a validation mean F1 score of 0.8621, outperforming over 1600 competing architectures, including manually constructed CNNs, AlexNet, ResNet variations, Inception modules, and models generated by alternative algorithms such as Bayesian Optimization, Particle Swarm Optimization and the Genetic Algorithm. These results emphasize the capacity of Differential Evolution for autonomous network design and hyperparameter tuning when training memory-efficient models. • Design of memory-efficient multimodal models through novel Differential Evolution algorithm. • Application of Word2Vec matrix translations for convolutional processing. • Benchmarked algorithmic performance against leading competitive algorithms. • Competitive performance of Differential Evolution for NAS and hyperparameter tuning.