Efficient Bangla OCR via Sequence-Level Knowledge Distillation

The objective of Optical Character Recognition (OCR) is to convert handwritten and printed information into a digital version. While OCR has made tremendous advancements in recent years due to recurrent neural networks and the LSTM architecture, few systems exist to recognize Bengali characters in printed or handwritten documents, and those that do exist can be computationally expensive. Knowledge distillation (KD) is an excellent method for transferring knowledge from a complex teacher model to a simpler student model. In KD, a loss function is employed to train the student model to imitate the behavior of the teacher model by matching the soft targets created by the teacher model with the student model’s predictions. In this study, a novel loss function is proposed for sequence-level knowledge distillation in CTC-based Bangla OCR. Combining KL divergence and Mean Squared Error (MSE), our loss function transfers knowledge from the teacher model to the student model. The proposed strategy seeks to improve the Character Recognition Rate (CRR), Word Recognition Rate (WRR), and F1 macro score for the student model’s minor classes. Compared to the student model trained using simply KL divergence, our strategy delivers considerable gains in CRR, WRR, and F1 macro scores.