Diferenciando o Não Diferenciável: Investigando problemas na diferenciação da ReLU no treinamento de modelos de Aprendizado Profundo

Activation functions are necessary for deep learning models to beable to represent non-linear relationships. Their derivative is requiredduring training, however, many non differentiable activationfunctions are commonly used in neural networks, such as the RectifiedLinear Unit (ReLU) and its variants. This paper discusses theimpact of non differentiability on activation functions during thetraining phase, and how these functions compare to differentiablealternatives. To analyse this problem, we trained neural networksin an image classification problem using various activation functions.We showed that non-differentiable points occur rarely duringtraining, especially in deep models, and have little to no negativeimpact in a model’s performance.