Data-free binary neural network with image heterogeneity

Binary neural networks (BNNs) are widely used in image and video processing tasks because they can greatly reduce memory and accelerate inference speed to deploy in edge scenes. However, the deployment of BNN still faces the problem that the original data is not available. Data-free adversarial distillation (DFAD) focuses on solving model training problems in data-free scenarios. Therefore, combining BNN with DFAD provides a potential solution for lightweight model deployment in data-free scenarios. We introduce BNN into the DFAD framework for the first time, called Data-free binary neural network (DFBNN), but this combination faces challenges. Due to the information loss of BNN, the samples generated by the generator often lose category information, and the heterogeneity between categories is high, which leads to the decrease of learning performance of BNN student model. In order to cope with this challenge, we propose the category information entropy maximization loss to constrain the probability distribution of the samples generated in the same batch. In addition, the intra-class feature clustering constraint is proposed. By limiting the cosine distance of the same category features of the generated samples, the inter-class heterogeneity is ensured while focusing on solving the problem of low intra-class heterogeneity. A large number of experiments show that DFBNN is feasible and superior, and provides an effective solution for lightweight model deployment in data-free scenarios.