Improving Dense Retrieval on Domain Adaptation and Decontextualization

Dense retrieval embeds and matches text in a continuous vector space, enabling effective and efficient semantic search. However, it requires large amounts of training data, which incurs substantial costs to achieve its optimal effectiveness. Most importantly, even after training on massive data, previous work finds that dense retrievers severely degrade under a domain shift. This issue greatly limits the applicability of dense retrieval in various scenarios. To address this challenge, we first propose an unsupervised sentence embedding approach based on pre-trained Transformers and Sequential Denoising AutoEncoder (TSDAE). TSDAE requires only unlabeled texts from the target domain to obtain an embedding model for computing the similarity of text pairs in that domain. It also serves as a strong domain adaptation approach when applied before supervised fine-tuning on the data from the source domain. The experiment results show that unsupervised learning with TSDAE achieves up to 93.1% of the performance of in-domain supervised approaches; domain adaptation with TSDAE outperforms the previous best approach by up to 4.5 points of average precision. In parallel to the pre-training fashion of TSDAE, we also propose a post-training approach, Generative Pseudo Labeling (GPL) for unsupervised domain adaptation based on data augmentation. GPL utilizes a query generator to generate raw training data for the target domain and labels the data with relevance scores for the final training. We show that GPL can outperform an out-of-the-box State-of-the-Art (SotA) dense retrieval approach by up to 9.3 points in nDCG@10. When combined with TSDAE, the retrieval performance can be further enhanced by 1.4 points in nDCG@10 across six tasks. Besides generalizing to new domains, dense retrieval is also limited in encoding short passages independently without considering the context of the associated documents. In our error analysis on a web-search dataset, we find that more than half of the errors of the SotA retrievers are due to missing document context. Driven by this observation, we propose a new task, Document-Aware Passage Retrieval (DAPR) and build a benchmark of the same name with datasets from heterogeneous domains. In the DAPR task, a passage is retrieved by considering its associated document context. We experiment with multiple approaches to extend the SotA retrievers by incorporating document context: (1) hybrid retrieval with BM25 (the widely used retrieval system based on simple lexical matching) and (2) contextualized passage representations that inform the passage representation with document context. We find that while hybrid retrieval performs strongest on the mixture of the easy and the hard examples, it completely fails on the hard examples that require an understanding of document context. On the other hand, contextualized passage representations (e.g. prepending document titles) achieve good improvement on these hard examples, but overall they also perform rather poorly. The benchmark facilitates future research on developing and comparing retrieval systems for this new task.