2P-BEnc: A two-phase information retrieval and ranking system based on the BERT encoder

Information retrieval methods have been advanced by the development of Natural Language Understanding (NLU). The development of deep neural networks was a key driver in the creation of effective Language Models (LM: statistical models trained to understand and generate human language), which significantly enhanced document retrieval. Even with these improvements, the current system for finding and ranking information faces significant challenges. In this work, we are focused on three of them. These problems are the system’s maximum passage length processing capacity, the cost of making inferences, and integration of information across passages. The proposed 2P-2P- can handle longer and more passages while keeping the cost of making inferences low. Also, 2P-2P- uses more than one passage to make the ranked list of retrieval results. This way, 2P-BEnc stores information from different passages and improves ranking performance. Mechanism: 2P-BEnc achieves this through a cascaded encoder design: (1) Sentence-level processing bypasses BERT’s (Bidirectional Encoder Representations from Transformers: a transformer-based language model that establishes contextual relationships through attention mechanisms) token limits by treating sentences as atomic units, (2) Offline precomputation of sentence vectors slashes inference costs (Computational resources required during prediction phase) by 300 × , (3) Cross-passage attention in Phase 2 aggregates contextual signals across documents. When the performance of the proposed system is compared to benchmarking datasets that are available to the public, the results are competitive and encouraging. On the MS-MARCO dataset, the proposed 2P-BEnc model achieved 38.6 MRR@10, which is 5.75 % higher than the BERT Large model. On the TREC-CAR dataset, the proposed 2P-BEnc model achieved 35.4 MAP@1000, which is 5.67 % higher than BERT Large . Aside from that, the 2P-BEnc performed as well as or better than BERT L a r g e on other benchmark datasets (Robust04, WikiPassageQA, and WikiQA).