Modern small networks for image classification. Feature analysis

With the increasing demand for deploying deep learning models on resource-constrained devices, such as smartphones, IoT sensors, and edge computing platforms, the need for efficient convolutional neural networks (CNNs) has become paramount. This paper offers a comprehensive review of several state-of-the-art lightweight CNN architectures designed to address these challenges by reducing computational complexity and memory usage, while maintaining competitive performance in image classification tasks. Key architectures reviewed include MobileNets, ShuffleNet, DiceNet, and ESPNet, each of which employs distinct strategies to optimize network efficiency. MobileNets introduce the concept of depthwise separable convolutions, which decompose the standard convolution operation into a depthwise convolution and a point-wise convolution (1x1). This drastically reduces the number of parameters and computations compared to traditional convolutions. ShuffleNet, on the other hand, leverages group convolutions and channel shuffling to enhance efficiency, allowing feature maps to be split and recombined, which reduces computational cost without significantly compromising accuracy. DiceNet builds upon these concepts by introducing multi-branch architecture with different dilation rates to capture features at multiple scales, enhancing both accuracy and efficiency in low-resource environments. ESPNet employs efficient spatial pyramidal structures, along with point-wise convolutions, to handle diverse spatial features at different scales while being highly computationally efficient. Despite these advancements, a common bottleneck across these architectures is the reliance on point-wise (1x1) convolutions, which, while more efficient than standard convolutions, still contribute significantly to the overall computational cost, particularly in deeper layers of the network. Furthermore, filter sizes are often optimized for performance in a cloud-based setting but may not be ideal for edge environments where computational and energy efficiency are crucial. We see the potential in changing filter sizes in some layers to 2x2 which is the smallest possible filter for spatial information extraction. Also it worth paying attention to the way the information is spread across channels as well as how the channels number is formed by replacing 1x1 convolution with a generic but yet predictable mathematical operation.