Lightweight anomaly detection via knowledge distillation from ensemble teachers

This paper proposes a knowledge distillation-based anomaly detection method that employs a heterogeneous Isolation Forest ensemble as the teacher and a lightweight neural network as the student. The student learns the teacher’s soft labels by minimizing the Kullback–Leibler (KL) divergence. Evaluated on three categories from the MVTec 3D-AD dataset (bagel, carrot, and cookie), the teacher achieves a macro-average area under the receiver operating characteristic curve (AUROC) of 0.6406, whereas the student, with 2.010 million parameters in the multilayer perceptron (MLP) head and an approximately 0.051 ms detector-head inference time per image (both excluding the ResNet-50 backbone), achieves a macro-average AUROC of 0.5937 and delivers an approximately 11.6-fold detector-head inference speedup over the teacher ensemble. Compared with representative deep learning baselines (RD4AD and PatchCore), the proposed method achieves moderate accuracy with lower detector-head latency and a smaller detector-head model footprint (excluding the backbone). These results suggest that knowledge distillation can serve as an effective compression technique, transferring ensemble knowledge to a lightweight student with a measurable accuracy trade-off. The benefit of distillation is category-dependent: the student outperforms the teacher on one category but exhibits performance degradation on the others. Moreover, the No-KD ablation (student trained without KD) achieves a higher macro-average AUROC (0.6057 vs. 0.5937), indicating that soft-label distillation is a viable compression strategy rather than a universal performance enhancer. The conditions under which distillation improves or degrades performance are discussed.