Engine Availability Decay in Keyless Metasearch: a small latency and reliability study
Metasearch front-ends let an agent query many upstream engines behind one endpoint, and the "keyless" ones are attractive because they need no credentials. What is rarely measured is how quickly that breadth collapses once traffic is sustained. We instrumented two profiles of a self-hosted, SearXNG-compatible metasearch service (RustXNG 0.32.0) on a single machine: a conservative profile of 34 keyless providers, and an "extended" profile that adds three scraped engines. Across 30 timed queries and a 40-request reliability pass we find (i) the extended profile costs roughly 1.8× the median latency of the conservative one (1012 ms vs 569 ms, medians over all classes), (ii) coverage is nonetheless concentrated in three providers that supply 69% of all returned results, and (iii) six engines stop answering entirely within the observation window, alternating between resting ("no results lately", 120 occurrences) and suspended (CAPTCHA, 80; unexpected error, 39). The practical implication is that a keyless profile is not a stable dependency: availability decays inside a single session, while latency is paid up front on every query. 1. Introduction Self-hosted metasearch — SearXNG and its Rust re-implementation RustXNG — aggregates results from external engines and ranks them locally [1,2]. For agent workloads the appeal is twofold: one HTTP surface instead of many, and profiles that can be widened without provisioning API keys. Breadth, however, is bought from engines that were designed for human browsing, several of which rate-limit, CAPTCHA, or silently return nothing under programmatic load. A planner that widens a profile during an incident may therefore be trading a measurable cost (latency on every request) for coverage that disappears part-way through the session.
Authors
- Jorden Smith (ORCID: https://orcid.org/0009-0008-1992-8083)
Institutions
- National University of Singapore (SG)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-19
- DOI
- https://doi.org/10.5281/zenodo.24382665
- Primary Topic
- Information Retrieval and Search Behavior
- Type
- preprint