Engine Availability Decay in Keyless Metasearch: a small latency and reliability study

Metasearch front-ends let an agent query many upstream engines behind one endpoint, and the "keyless" ones are attractive because they need no credentials. What is rarely measured is how quickly that breadth collapses once traffic is sustained. We instrumented two profiles of a self-hosted, SearXNG-compatible metasearch service (RustXNG 0.32.0) on a single machine: a conservative profile of 34 keyless providers, and an "extended" profile that adds three scraped engines. Across 30 timed queries and a 40-request reliability pass we find (i) the extended profile costs roughly 1.8× the median latency of the conservative one (1012 ms vs 569 ms, medians over all classes), (ii) coverage is nonetheless concentrated in three providers that supply 69% of all returned results, and (iii) six engines stop answering entirely within the observation window, alternating between resting ("no results lately", 120 occurrences) and suspended (CAPTCHA, 80; unexpected error, 39). The practical implication is that a keyless profile is not a stable dependency: availability decays inside a single session, while latency is paid up front on every query. 1. Introduction Self-hosted metasearch — SearXNG and its Rust re-implementation RustXNG — aggregates results from external engines and ranks them locally [1,2]. For agent workloads the appeal is twofold: one HTTP surface instead of many, and profiles that can be widened without provisioning API keys. Breadth, however, is bought from engines that were designed for human browsing, several of which rate-limit, CAPTCHA, or silently return nothing under programmatic load. A planner that widens a profile during an incident may therefore be trading a measurable cost (latency on every request) for coverage that disappears part-way through the session.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-19
DOI
https://doi.org/10.5281/zenodo.24382665
Primary Topic
Information Retrieval and Search Behavior
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Engine Availability Decay in Keyless Metasearch: a small latency and reliability study

Jorden Smith
Zenodo (CERN European Organization for Nuclear Research)
Information Retrieval and Search Behavior
preprint

Engine Availability Decay in Keyless Metasearch: a small latency and reliability study

Jorden Smith
preprint en

Abstract

Metasearch front-ends let an agent query many upstream engines behind one endpoint, and the "keyless" ones are attractive because they need no credentials. What is rarely measured is how quickly that breadth collapses once traffic is sustained. We instrumented two profiles of a self-hosted, SearXNG-compatible metasearch service (RustXNG 0.32.0) on a single machine: a conservative profile of 34 keyless providers, and an "extended" profile that adds three scraped engines. Across 30 timed queries and a 40-request reliability pass we find (i) the extended profile costs roughly 1.8× the median latency of the conservative one (1012 ms vs 569 ms, medians over all classes), (ii) coverage is nonetheless concentrated in three providers that supply 69% of all returned results, and (iii) six engines stop answering entirely within the observation window, alternating between resting ("no results lately", 120 occurrences) and suspended (CAPTCHA, 80; unexpected error, 39). The practical implication is that a keyless profile is not a stable dependency: availability decays inside a single session, while latency is paid up front on every query. 1. Introduction Self-hosted metasearch — SearXNG and its Rust re-implementation RustXNG — aggregates results from external engines and ranks them locally [1,2]. For agent workloads the appeal is twofold: one HTTP surface instead of many, and profiles that can be widened without provisioning API keys. Breadth, however, is bought from engines that were designed for human browsing, several of which rate-limit, CAPTCHA, or silently return nothing under programmatic load. A planner that widens a profile during an incident may therefore be trading a measurable cost (latency on every request) for coverage that disappears part-way through the session.

Zenodo (CERN European Organization for Nuclear Research)
National University of Singapore (SG)
Information Retrieval and Search Behavior
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon