Deep Contrastive Active Learning for Out-of-domain Filtering in Dialog Systems

Task-oriented dialog systems have shown to foster effective human-chatbot collaborations for accomplishing goal-specific tasks through intent classification. In a real-world setting, collecting and training over user intents incurs a labeling-cost challenge for human annotators. While existing human-AI collaborative approaches such as active learning (AL) can properly resolve such labeling-cost challenges, most existing AL algorithms assume the unlabeled pool has similar distributions as the in domain (IND) training set. To address the conflict between AL and out-of-domain (OOD) data samples, we present Deep Contrastive Active Learning (DeCAL), a deep novel AL framework that uses contrastive learning techniques for query intent classification in task-oriented dialogs. DeCAL features an acquisition function that filters OOD samples by computing a distance-based confidence score over unlabeled samples using their neighboring features. To validate DeCAL, we compare against deep AL baselines via the performance of acquired IND/OOD samples and using the classification accuracy metric. Experimental results on benchmark datasets demonstrate De-CAL outperforms deep AL baseline algorithms on acquired OOD by 14%, while simultaneously showing competitive performance on IND accuracy.