Data Reliability Enhanced Prediction for Recommendation System: A Case Study on Named Entity Recognition

Prediction reliability of deep learning based systems allows users to confirm if the prediction is reliable to real applications, which is the key to the success of recommendation systems. Current research on deep learning based recommendation systems focused on estimating model reliability, but seemed to be missing data contributions to the prediction reliability. This paper proposed a novel framework to estimate the prediction reliability for deep learning-based methods through combining the model reliability and the data reliability. The proposed framework has been validated in a case study based on named entity recognition (NER) that is from an Intuit recommendation task. It employed two NER datasets: WNUT and GMB to examine detailed performance of the proposed framework, where, specifically, we proposed a novel evaluation metric to comprehensively evaluate the performance. Experimental results demonstrated that, compared to model reliability only, combining data reliability with model reliability will significantly improve performance, as well as enhance the prediction interpretability.