Unsupervised Semantic Retrieval via Neural Estimation of Pointwise Mutual Information

Semantic retrieval task aims to identify documents in a retrieval corpus that are semantically related to a query. Existing methods rely heavily on large-scale annotated training data, which are expensive to obtain. To relieve the reliance on annotated data, this paper investigates the semantic retrieval problem in an unsupervised setting, where a corpus of candidate documents is given without accessing the training data. In this setting, the utilization of deep learning techniques may be limited by the insufficient consideration of word-level correlations. To address this limitation, this study seeks to solve the unsupervised semantic retrieval problem by Pointwise Mutual Information (PMI). Specifically, we theoretically show that the solutions for semantic retrieval based on maximum a posteriori and maximum likelihood are equivalent to the maximumization of PMI, enabling us to tackle unsupervised semantic retrieval by PMI estimation. However, existing PMI estimation with the incidence matrix may suffer from the matrix-sparsity problem, which may cause the empirical distribution to be discordant with the ground truth distribution and lead to inaccurate PMI estimation. To tackle this problem, we propose a new unsupervised PMI estimation framework, i.e. UPMI, that leverages negative sampling to reformulate PMI estimation as a binary classification problem, whose optimal solution is exactly the PMI. We then employ neural networks equipped with attention mechanisms to realize the PMI function in the UPMI framework. Empirical studies on 13 benchmark datasets of three unsupervised semantic retrieval tasks confirm the effectiveness of our proposed model.