Efficient Attention in Partially Relevant Video Retrieval: A Benchmarking Study on Accuracy-Efficiency Trade-Offs

The quadratic complexity of standard attention mechanisms remains a critical bottleneck for transformer-based models in Partially Relevant Video Retrieval (PRVR), particularly when processing long video sequences. This paper presents the first systematic study of efficient attention mechanisms for PRVR, evaluating three representative approaches—Linear Attention, Agent Attention, and FFT Attention—on the Charades-STA benchmark. We introduce a novel Harmonic Performance-Efficiency Score (HPES) to quantify the trade-offs between retrieval accuracy and computational overhead, addressing the limitations of conventional single-metric evaluations. Experiments reveal that FFT Attention achieves the highest HPES (1.13), outperforming Softmax baseline with <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$1.3 \times$</tex> speedup and higher SumR, demonstrating its superiority in balancing global context modeling. Our findings provide a principled framework for selecting attention mechanisms in PRVR models.