A Hybrid Architecture for Out of Domain Intent Detection and Intent Discovery

Intent Detection is a critical component of Natural Language Understanding (NLU) in task-oriented dialogue systems. These systems face two key challenges: handling Out of Scope (OOS) and Out of Domain (OOD) inputs that can lead to incorrect responses, and the resourceintensive process of creating labeled datasets for new intents. This paper addresses both challenges through a novel hybrid architecture. For OOD/OOS intent detection, we propose a Variational Autoencoder (VAE) that can distinguish between known and unknown intents while being robust to input data distribution variations. For intent discovery, we employ an unsupervised clustering approach enhanced by non-linear dimensionality reduction to identify coherent groups among OOD/OOS inputs. Our experimental results demonstrate robust performance across datasets in both English and Persian, with our hybrid architecture achieving strong F1 scores (>85%) in OOD detection and high clustering accuracy (>70%) compared to existing approaches.