Teacher privileged distillation: How to deal with imperfect teachers?

The paradigm of learning using privileged information leverages privileged features present at training time, but not at prediction, as additional training information. The privileged learning process is addressed through a knowledge distillation perspective: information from a teacher learned with regular and privileged features is transferred to a student composed exclusively of regular features. While most approaches assume perfect knowledge for the teacher, it can commit mistakes. Assuming that, we propose a novel privileged distillation framework with a double contribution. Firstly, a designed function to imitate the teacher when it classifies correctly and to differ in cases of misclassification. Secondly, an adaptation of the cross-entropy loss to appropriately penalize the instances where the student outperforms the teacher. Its effectiveness is empirically demonstrated on datasets with imperfect teachers, significantly enhancing the performance of state-of-the-art frameworks. Furthermore, necessary conditions for successful privileged learning are presented, along with a dataset categorization based on the information provided by the privileged features. • Necessary conditions for a successful privileged learning based on information theory. • New categorization of the datasets derived from the information offered by the privileged features. • A novel privileged learning framework, Teacher Privileged Distillation (TPD), to deal with imperfect teachers.