From Lambda to One: A Kafka-Compatible Streaming Lakehouse for Unified Stream and Analytical Workloads

Modern data architectures maintain separate systems for stream processing and analytical queries, leading to duplicated pipelines, operational complexity, and consistency challenges. We present Singularity, a Kafka compatible streaming lakehouse that unifies streaming and analytical workloads at the storage layer. By representing streams as versioned columnar datasets (Apache Lance on S3), Singularity enables Reactive Analytical Streaming (RAS): SQL queries execute over both historical and live streaming data and materialize results as Kafka topics. A single declarative SQL statement replaces both batch and streaming pipelines — historical backfills and real time updates are expressed identically. Existing Kafka producers and consumers migrate with zero code changes, while analytical engines query the same data directly without ETL. We describe the system architecture, execution semantics, and how unified storage eliminates data duplication, reduces latency, and ensures consistency betweenreal-time and historical computation.