Beyond Vector Similarity: A Two-Stage Safety-Aware Semantic Caching Architecture for Large Language Model Gateways

Large Language Model (LLM) gateways rely heavily on semantic caching to reduce inference latency and computational cost by reusing responses for semantically similar queries. However, conventional single-stage semantic caches that depend solely on dense vector cosine similarity are structurally vulnerable to false-positive cache hits. High vector similarity frequently fails to distinguish logical negations, antonym substitutions, subtle parameter shifts, and safety-critical boundary conditions—resulting in contextually incorrect or unsafe cached responses being served to users. This paper presents a two-stage safety-aware semantic caching architecture designed for production LLM gateways. Stage 1 performs fast approximate nearest-neighbor retrieval using dense vector embeddings (pgvector) to identify candidate cache entries above an initial similarity threshold. Stage 2 introduces a deterministic safety and intent verification gate that evaluates structural divergence, negation tokens, and policy boundaries before confirming a cache hit. By decoupling fast candidate retrieval from strict semantic equivalence verification, the proposed architecture significantly reduces false-positive cache hits on adversarial and edge-case prompts while maintaining low end-to-end latency overhead.