Retrieval for Semantic People Search

These days we have a large number of open-source embedding LLMs that can be leveraged in a bi-encoder architecture for retrieval. However, they do not perform very well when leveraged as is for people search because the (query, document) pairs encountered in people search are much more complex than the text pairs that these open-source embedding LLMs are trained on. In this paper we present our experiments, learnings and solution to the problem of retrieval for people search. Our solution involves query simplification, document simplification, fine-tuning of a 7B parameter embedding LLM, and compression of embeddings through Matryoshka learning.