An In-Cell Weight Update Scheme Using One-Hot Gradients for On-Chip Learning

In the context of edge AI, the embedded models need to be retrained on-chip. The more the backward path and the calculation the gradient (two important steps during learning) are optimized by literature, the more it becomes interesting to optimize weight update operation. This work proposes to encode the weight update gradient as a power of two. It is shown that this encoding method has a low impact on the learning capability for neural network configurations and it reduces the amount of weight update operations during training by a factor of 12.5 in the case of a feedforward neural network and by a factor of 9.5 in the case of a convolutional configuration. It is also compatible with transfer learning approaches. In addition, using in-memory computing, it is possible to get rid of the read operation required for the weight update operation and to reduce the amount of writing operation by 8, thereby reducing further the energy required for a weight update by another factor of 13.7. Finally considering the reduction of energy needed per update and the reduction of update operation done during training process, the energy needed for weight update can be reduced by 137 in the case of a convolutional architecture.