Reconciling Streaming Semantics with Batch-Oriented ML Inference: Architectural Patterns for Production-Scale Multi-Modal Candidate Retrieval

Production candidate-retrieval systems increasingly combine streaming ingestion with multi-modal embedding generation and vector-indexed search, yet the infrastructure components underlying each stage were not designed around a shared execution model. Stream processors such as Apache Flink favour record-at-a-time or micro-batched dataflow with strict checkpointing guarantees, while machine learning inference services achieve favourable throughput only when requests are batched, and vector databases such as Milvus assume comparatively infrequent mutation relative to read volume. This paper documents two recurring friction points at these pipeline boundaries: the granularity mismatch between per-record streaming calls and batch-efficient inference serving, and the write amplification created by high-churn item updates against approximate nearest neighbour indexes. Two architectural patterns addressing each friction point are presented: adaptive micro-batching with checkpoint-integrated buffering for the streaming-to-inference boundary, and lazy invalidation backed by an external version registry for the vector-mutation boundary. Their interaction is examined, along with the workload conditions under which each pattern's trade-offs are acceptable. The patterns are offered as a reusable analytical framework grounded in systems literature and practitioner experience, and their applicability depends on workload freshness and latency tolerances.

Authors

Publication Details

Journal
East African Journal of Information Technology
Published
2026-11-13
DOI
https://doi.org/10.37284/eajit.9.3.6275
Primary Topic
Advanced Database Systems and Queries
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Reconciling Streaming Semantics with Batch-Oriented ML Inference: Architectural Patterns for Production-Scale Multi-Modal Candidate Retrieval

Sandeep Ayloo
East African Journal of Information Technology
Advanced Database Systems and Queries
article

Reconciling Streaming Semantics with Batch-Oriented ML Inference: Architectural Patterns for Production-Scale Multi-Modal Candidate Retrieval

Sandeep Ayloo
article en

Abstract

Production candidate-retrieval systems increasingly combine streaming ingestion with multi-modal embedding generation and vector-indexed search, yet the infrastructure components underlying each stage were not designed around a shared execution model. Stream processors such as Apache Flink favour record-at-a-time or micro-batched dataflow with strict checkpointing guarantees, while machine learning inference services achieve favourable throughput only when requests are batched, and vector databases such as Milvus assume comparatively infrequent mutation relative to read volume. This paper documents two recurring friction points at these pipeline boundaries: the granularity mismatch between per-record streaming calls and batch-efficient inference serving, and the write amplification created by high-churn item updates against approximate nearest neighbour indexes. Two architectural patterns addressing each friction point are presented: adaptive micro-batching with checkpoint-integrated buffering for the streaming-to-inference boundary, and lazy invalidation backed by an external version registry for the vector-mutation boundary. Their interaction is examined, along with the workload conditions under which each pattern's trade-offs are acceptable. The patterns are offered as a reusable analytical framework grounded in systems literature and practitioner experience, and their applicability depends on workload freshness and latency tolerances.

East African Journal of Information TechnologyVol. 9(3)
Advanced Database Systems and Queries
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon