Dataset distillation with stochastic neural networks

Dataset distillation aims to synthesize tiny and high-fidelity data that contains the most important information of a given target dataset. Recent studies primarily used gradient-matching based methods to attain practical performance. Observing that proper randomness in gradient matching is meaningful for the training of condensed datasets, we propose to distill the dataset with stochastic neural networks. To this end, we propose four types of stochastic neural networks to introduce stochasticity: data-independent noise, data-dependent noise, vanilla dropout, and nested dropout. Additionally, we introduce crucial methods for estimating uncertainty to manage the inherent randomness in the distillation process. The final experiments demonstrate that integrating stochastic neural networks and uncertainty estimation effectively enhances the quality and speed of dataset distillation in a plug-and-play way.