The Impact of Neural Network Architecture on Generalization: A Review of Depth, Width, and Regularization Techniques

This paper investigates the impact of increasing the depth and width of convolutional neural networks (CNNs) on their generalization performance across image classification tasks. Experiments were conducted using PyTorch on two datasets of varying complexity: MNIST (simple) and CIFAR-10 (complex). A variety of CNN architectures were trained with different depths and widths, and regularization techniques including dropout and L2 weight decay were applied to analyze their effects on overfitting. Results indicate that shallow networks are sufficient for achieving high accuracy on MNIST, while deeper or wider networks yield significant performance gains on CIFAR-10. However, high-capacity models are more prone to overfitting without appropriate regularization. Techniques such as dropout and L2 regularization were found to consistently improve generalization, particularly in deeper architectures. These findings underscore the importance of balancing model complexity and regularization, especially when dealing with datasets of differing size and variability.