Enhancing Binary Neural Network with Response Deep Supervision

In recent years, binary neural networks (BNNs) have achieved significant success in resource-constrained scenarios. However, compared to full-precision networks, BNNs tend to have lower accuracy, which negatively impacts their application on complex tasks. To address this, we propose a novel BNN-RDS (Binary Neural Network with Response Deep Supervision) algorithm based on self-distillation and response deep supervision techniques. Firstly, we extend the backbone structure by auxiliary classifiers to transform features extracted from intermediate layers into logits response outputs. Then we effectively supervise intermediate layers using response-based deep supervision methods. Additionally, we introduce residual feature extraction modules and fully connected layers to enhance information propagation efficiency through residual connections and feature reuse mechanisms. This allows the network to better capture fine details and contextual information in images. Finally, we analyze how teacher layer selection differs between full-precision networks and BNNs during self-distillation. We propose utilizing the most accurate auxiliary classifier as the teacher layer in self-distillation and it ensures mutual benefit among multiple shallow classifiers to avoid negative interactions. Experimental results on CIFAR-I0 and ImageNet datasets show that the BNN-RDS method enhances the performance of BNNs. Furthermore, experiments on plant pathology datasets validate its effectiveness for identifying plant leaf diseases in agricultural domains.