Image Distillation with the Machine-Learned Gradient of the Loss Function and the K-Means Method

Image distillation is becoming a hot area of machine learning (ML) research because it minimizes the computer resources used for training of convolutional neural networks (CNNs). In this paper, we propose a novel method for image distillation that learns the gradient of the loss function after training a CNN with the original training image. It then modifies the images using the learned gradient and applies K-means clustering to split the modified images into clusters, in a class-wise fashion, and select their centroids as distilled images. We evaluated the capabilities of our method on MNIST, Fashion-MNIST, CIFAR-10, and the Oral Cancer Image Database (OCID) using three CNN models (Baseline, ResNet, and LeNet). The results show that when training the classifiers (CNN) with distilled images, we obtained accuracies of 99.93% for MNIST, 99.46% for Fashion-MNIST, 99.05% for CIFAR-10, and 91.8% for OCID, which are comparable with the results after training with the entire training set. Comparison of our distillation method with existing contemporary methods such as KIP and RTP showed the supremacy of the former even if trained with fewer epochs. A bottleneck of our method is that it works with gradient-based optimizers.