An NLP Approach to Efficient Duplicate Question Detection using Neural Networks and TF-IDF

Duplicate question pair detection (DQPD) is one of the significant factors influencing the user satisfaction in the NLP based platforms like Quora. With the increase in user generated content, identifying duplicate questions is also necessary. In this work, a new approach for DQPI is introduced and based on neural networks and TF-IDF features. For this experiment, we use Quora Question Pairs dataset obtained from Kaggle and our model achieves an impressive accuracy of 96.71% To handle the extensive datasets required for training, we utilize specialized data generation techniques. This work advances the field of Natural Language Processing by: Establishing the efficacy of introducing neural networks along with TF-IDF for DQPI. Making the training more memory efficient by using custom data generators. Increasing the Accuracy to detect Duplicate Question Pairs on a Realistic Dataset. It is envisioned that this work will be useful in following up on text classification with the neural network techniques in the NLP.