Leaky Dropout: Balancing Performance and Overfitting

Dropout layers are widely used in neural networks for regularization, but can reduce performance in some cases. We present Leaky Dropout, a generalization of dropout that scales down a random selection of inputs at each layer by a hyperparameter-determined factor, rather than silencing them. Dropout, Gaussian Dropout and our novel Leaky Dropout were analyzed and compared using out-of-sample performance and with respect to overfitting. Leaky dropout outperformed the other two methods on out-of-sample data, with little to no overfitting, and outperformed baseline models with no dropout in most cases. The performance and training were inversely related to overfitting and dependent on the scaling factor and the dropout rate. On small training sets and complex models, Leaky Dropout served as a better sort of regularization than the other two methods.