Learning Discrete Identifiers and Dense Vectors for Generative Retrieval

Generative retrieval presents a promising approach to information retrieval, streamlining both indexing and retrieval processes through end-to-end optimization. This method typically involves assigning a unique identifier to each document, with the retrieval goal being the generation of the correct document identifier in response to a query. Although generative retrieval has demonstrated empirical success in various tasks, designing an effective document identifier remains a challenge. Previous studies have either depended excessively on one-to-one discrete identifiers, leading to increased retrieval latency and loss of semantics in documents or have used retrieval-agnostic dense document identifiers, which can hinder performance. To this end, we propose to integrate the benefits of generative retrieval and dense retrieval using an encoder-decoder-based pre-trained language model. Particularly, the decoder, i.e., the discrete identifier, functions as a coarse retriever, effectively reducing the retrieval space in an end-to-end manner. As a complement, the encoder, i.e., the dense vector, serves as a fine-grained retriever, efficiently and precisely ranking documents in a condensed space. Accordingly, we introduce a three-stage end-to-end learning framework that optimizes identifiers and vectors. Extensive experiments reveal that the proposed method exceeds the current models in terms of effectiveness and time efficiency, across both small and larger corpus sets.