Reconciling Streaming Semantics with Batch-Oriented ML Inference: Architectural Patterns for Production-Scale Multi-Modal Candidate Retrieval
Production candidate-retrieval systems increasingly combine streaming ingestion with multi-modal embedding generation and vector-indexed search, yet the infrastructure components underlying each stage were not designed around a shared execution model. Stream processors such as Apache Flink favour record-at-a-time or micro-batched dataflow with strict checkpointing guarantees, while machine learning inference services achieve favourable throughput only when requests are batched, and vector databases such as Milvus assume comparatively infrequent mutation relative to read volume. This paper documents two recurring friction points at these pipeline boundaries: the granularity mismatch between per-record streaming calls and batch-efficient inference serving, and the write amplification created by high-churn item updates against approximate nearest neighbour indexes. Two architectural patterns addressing each friction point are presented: adaptive micro-batching with checkpoint-integrated buffering for the streaming-to-inference boundary, and lazy invalidation backed by an external version registry for the vector-mutation boundary. Their interaction is examined, along with the workload conditions under which each pattern's trade-offs are acceptable. The patterns are offered as a reusable analytical framework grounded in systems literature and practitioner experience, and their applicability depends on workload freshness and latency tolerances.
Authors
- Sandeep Ayloo
Publication Details
- Journal
- East African Journal of Information Technology
- Published
- 2026-11-13
- DOI
- https://doi.org/10.37284/eajit.9.3.6275
- Primary Topic
- Advanced Database Systems and Queries
- Type
- article
- Field-Weighted Citation Impact
- 0.00