Optimizing Approximate Nearest Neighbour-Based Dense Retrieval for Efficient Passage Ranking with Query Expansion

With the exponential growth of digital information, efficient and accurate passage retrieval remains a crucial challenge in information retrieval systems.This paper presents an optimized Approximate Nearest Neighbor-Based Dense Retriever for Passage Ranking with Query Expansion (ANN-DRP-QE), designed to enhance retrieval effectiveness while significantly reducing latency and memory usage.The model integrates query expansion techniques to refine query representations, improving retrieval performance across diverse information needs.The ANN-DRP-QE model is evaluated using the MS MARCO Passage Ranking Dataset, where its performance is compared against BM25 (a sparse lexical retrieval baseline), ColBERT (a dense multi-vector retrieval model), and a Standard ANN-Based Dense Retriever.After optimization, ANN-DRP-QE achieves the highest ranking performance with MRR@10 of 0.420, Recall@10 of 0.765, and nDCG@10 of 0.445, surpassing ColBERT (0.360, 0.728, 0.412), BM25 (0.187, 0.685, 0.372), and the Standard ANN-Based Retriever (0.342, 0.710, 0.401).Additionally, major efficiency enhancements, including HNSW indexing, Float16-based embeddings, FAISS dot-product search, batch query encoding, MiniLM embeddings, parallel query processing, and query expansion, drastically reduce latency to ~0.02-0.1s(compared to 0.015s for ColBERT, 0.3-0.6sfor Standard ANN, and 1.3s for BM25) while lowering memory consumption to ~0.004MB (matching ColBERT and over 100x smaller than BM25's 55.26MB).Finally, the results establish ANN-DRP-QE as the most efficient and scalable retrieval solution, achieving a superior balance between ranking accuracy, speed, and memory efficiency.The model presents a highly effective alternative for real-time passage ranking in AI-driven search engines and large-scale document retrieval systems.