Analyzing of Positive and Negative Prototypes based on the ±ED-WTA Method

The recently introduced method, ED-WTA, for each output neuron of a class in the last layer of neural networks, obtain a pair of positive and negative prototypes.The functionality of a neuron is explained based on the difference in the Euclidean distance of a sample from these two prototypes.An astonishing point in this method is the great similarity of positive and negative prototypes for each neuron at the end of training.The authors of [2] have claimed that the reason for this extreme similarity is the formation of the negative prototype of a class with samples of other classes, which are very similar to the positive prototype of that class.In this paper, it is demonstrated that not only does the negative prototype get close to the positive prototype of a class, but a positive prototype is also formed by samples that are far from the center of that class.The new finding about this great similarity shows that each neuron in the softmax layer, like SVMs, makes decisions based on the near-boundary samples.The theoretical analysis is examined in detail and experimental results on MNIST, FERET, and Fashion-MNIST show the correctness of the claims made.