Why NoSQL Benchmarks Fail in AI Era: A Practical Validity-Aware Workflow for Trustworthy Data Store Evaluation

Selecting a data-management subsystem is a consequential decision in data-intensive applications, including AI pipelines that rely on low-latency storage. Such decisions are often informed by lightweight, application-specific benchmarks whose measurement validity is rarely examined. This paper presents a validity-aware benchmarking workflow for comparing NoSQL data stores under application-representative CRUD workloads. The workflow incorporates a final validation stage that checks whether measured operations exhibit the expected behaviour before performance comparisons are made. The approach is demonstrated using a key-value store (Aerospike) and a document store (MongoDB). The validity checks identified two operations that performed no actual database work. MongoDB’s idiomatic read operation returned a cursor that was never iterated, causing reads to appear approximately 130 times faster for single-record workloads and 1000 times faster for bulk workloads than equivalent Aerospike reads. In addition, a MongoDB bulk update matched no documents because update keys were constructed incorrectly. Further validity threats included a missing secondary index, unequal bulk workloads, and a first-run warmup effect. Consequently, the remaining write measurements are reported descriptively, together with confidence intervals and run-level results. The study does not establish which engine delivers superior performance. Instead, it demonstrates how seemingly reasonable application-specific benchmarks can generate misleading conclusions and argues for an explicit validity-assessment stage, supported by a threat model, before benchmark results are used to inform subsystem selection and architectural decision-making.