Analog CMOS Implementation for Swish Activation Function

In analog computation, there are very few types of activation functions. In this study, we created a circuit that generates the difference between the Swish and the ReLU activation function, and aimed to mimic the behavior of the Swish function by combining ReLU and the difference. We obtained outputs close to the difference between the swish function and the ReLU function in simulation. We also confirmed that the sum of ReLU and the output works as activation function in neural networks on software.