Improving Neural Networks Dropout Using An Enhanced Weights Scaling

Dropout is a powerful regularization technique that successfully mitigates overfitting across a wide range of neural network architectures and applications, from computer vision to natural language processing and more. Dropout consists of dropping (i.e., zeroing) a random fraction of the layer’s nodes and then scaling the weights according to the complement probability of the dropout rate. This paper proposes an enhanced weight scaling and uses a simulation study over two benchmark datasets to demonstrate its improved accuracy over the standard approach.