Do Retrieval Ranking Models Capture Event Order? A Study of Timeline-Constrained Retrieval

Semantic retrieval models aim to match documents to a query's "meaning", yet it remains unclear whether they successfully encode temporal relations between events in their ranking mechanisms. We investigate this question by studying timeline-constrained document retrieval, where a query is an ordered sequence of natural-language event statements while a relevant document should match those events in a compatible temporal progression. We then test whether widely used dense retrievers and neural rerankers capture such constraints, and whether explicitly preserving temporal order improves ranking, using a controlled setting over a novel focused evaluation collection. Experiments show that state-of-the-art semantic retrievers and rerankers provide only limited improvements over lexical retrieval, suggesting that their semantic matching capabilities do not reliably capture temporal constraints. We then compare a timeline-aware method with both semantic matching and standard rerankers and find that it yields substantial improvements. This demonstrates that considering temporal order can provide a strong ranking signal in this setting, suggesting future research on improving temporal modeling in semantic retrievers.