An Adaptive ANN Indexing and Parameter-Tuning Framework for SLA-Constrained Retrieval-Augmented Generation (RAG) Microservices in Relational Vector Databases

Enterprise Retrieval-Augmented Generation (RAG) architectures increasingly consolidate operational relational metadata and high-dimensional vector embeddings within unified relational database management systems (RDBMSs) to eliminate cross-system synchronization latency and preserve transactional ACID guarantees. However, executing Approximate Nearest Neighbor (ANN) searches—specifically over Hierarchical Navigable Small World (HNSW) proximity graphs and Inverted File with Flat Compression (IVFFlat) partitions—inside stateless, multi-tenant RAG microservices exposes a severe multi-objective optimization bottleneck. Production retrieval pipelines must simultaneously satisfy strict 99th-percentile (p99) tail-latency Service Level Agreements (SLAs), operate within bounded relational shared-buffer memory pools, and guarantee high top-k semantic recall (Recall@k) under non-stationary query concurrency. Static, heuristically selected index topologies and runtime parameters frequently induce either catastrophic buffer-pool page thrashing when graph structures exceed resident memory or severe queueing-induced SLA violations during burst traffic. To address this fundamental systems challenge, this paper proposes an Adaptive ANN Indexing and Parameter-Tuning Framework (AAIPT) underpinned by a formal buffer-pool-aware analytical cost model and an M/G/c queueing-theoretic predictive admission controller. First, we derive a closed-form page-fault and service-time cost model that couples relational 8-KB buffer page layouts, HNSW random-walk pointer locality, and IVFFlat sequential scan characteristics to select optimal offline index topologies without inducing out-of-core thrashing. Second, we formulate online runtime hyperparameter modulation—governing graph candidate exploration depth (ef_search) and Voronoi cell probing (n_probe)—as a Lyapunov-damped constrained tail-latency minimization problem driven by real-time connection-pool utilization and service-time variance. Third, we conduct an extensive empirical evaluation across three large-scale benchmark workloads (d in {384, 768, 1536}, N up to 2 x 10^6 vectors, and arrival rates up to 300 QPS) comparing AAIPT against five static and heuristic baselines. Experimental results demonstrate that AAIPT reduces p99 tail latency by 38.4% to 52.1%, eliminates out-of-core buffer thrashing spikes of up to 310%, and reduces microservice SLA violation rates from 41.5% down to 0.8% while strictly preserving Recall@10 >= 0.95.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24220224
Primary Topic
Advanced Database Systems and Queries
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

An Adaptive ANN Indexing and Parameter-Tuning Framework for SLA-Constrained Retrieval-Augmented Generation (RAG) Microservices in Relational Vector Databases

Sudipta Mallikarjun, Fayaz Kakkalmeli
Zenodo (CERN European Organization for Nuclear Research)
Advanced Database Systems and Queries
article

An Adaptive ANN Indexing and Parameter-Tuning Framework for SLA-Constrained Retrieval-Augmented Generation (RAG) Microservices in Relational Vector Databases

Sudipta Mallikarjun, Fayaz Kakkalmeli
article en

Abstract

Enterprise Retrieval-Augmented Generation (RAG) architectures increasingly consolidate operational relational metadata and high-dimensional vector embeddings within unified relational database management systems (RDBMSs) to eliminate cross-system synchronization latency and preserve transactional ACID guarantees. However, executing Approximate Nearest Neighbor (ANN) searches—specifically over Hierarchical Navigable Small World (HNSW) proximity graphs and Inverted File with Flat Compression (IVFFlat) partitions—inside stateless, multi-tenant RAG microservices exposes a severe multi-objective optimization bottleneck. Production retrieval pipelines must simultaneously satisfy strict 99th-percentile (p99) tail-latency Service Level Agreements (SLAs), operate within bounded relational shared-buffer memory pools, and guarantee high top-k semantic recall (Recall@k) under non-stationary query concurrency. Static, heuristically selected index topologies and runtime parameters frequently induce either catastrophic buffer-pool page thrashing when graph structures exceed resident memory or severe queueing-induced SLA violations during burst traffic. To address this fundamental systems challenge, this paper proposes an Adaptive ANN Indexing and Parameter-Tuning Framework (AAIPT) underpinned by a formal buffer-pool-aware analytical cost model and an M/G/c queueing-theoretic predictive admission controller. First, we derive a closed-form page-fault and service-time cost model that couples relational 8-KB buffer page layouts, HNSW random-walk pointer locality, and IVFFlat sequential scan characteristics to select optimal offline index topologies without inducing out-of-core thrashing. Second, we formulate online runtime hyperparameter modulation—governing graph candidate exploration depth (ef_search) and Voronoi cell probing (n_probe)—as a Lyapunov-damped constrained tail-latency minimization problem driven by real-time connection-pool utilization and service-time variance. Third, we conduct an extensive empirical evaluation across three large-scale benchmark workloads (d in {384, 768, 1536}, N up to 2 x 10^6 vectors, and arrival rates up to 300 QPS) comparing AAIPT against five static and heuristic baselines. Experimental results demonstrate that AAIPT reduces p99 tail latency by 38.4% to 52.1%, eliminates out-of-core buffer thrashing spikes of up to 310%, and reduces microservice SLA violation rates from 41.5% down to 0.8% while strictly preserving Recall@10 >= 0.95.

Zenodo (CERN European Organization for Nuclear Research)
Gulbarga University (IN), Government of Karnataka (IN)
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon