Replication as Lineage Mechanism for Materialized Views in Lakehouse Architectures

Nowadays, the volume of data collected from online activity is continuously growing, starting from ecommerce websites, mobile applications or even IoT devices and sensors. All this data must be processed by more and more complex Big Data applications that gather and provide most useful information out of it. Consequently, the storage systems have evolved to keep up with all these changes, transitioning from traditional Data warehouses to Data Lakes and now to Lakehouses. All these applications consist of various transformations on stored data that result in another improved version of it, enriched with some new knowledge that brings value to the user. Based on the type of transformation, one can distinguish at least two types. The first can be simply executed on demand at runtime to get some insights from the collected data, such as the events count or some aggregated reports. The other type is more complex by its nature because it involves some complex processing of that data solely or in conjunction with other data. So, the result of such a transformation is both expensive and not feasible to be computed at runtime every time it is needed. Therefore, it must be persisted on storage once it is computed and then, kept up to date with the latest changes. This persisted computed version of the data is named materialized view, and multiple materialized views can be chained in data pipelines if the result of one transformation represents the input of the next one. Replication is the mechanism that adds lineage semantics to the materialized view transformation, independent of how many chained materialized views are kept, such as in the case of the bronzesilver-gold tables in the medallion architecture. Furthermore, this identifiability feature added to the materialized data facilitates both the propagation of all new incoming changes and the at-least-once processing semantics of the system in case of data reprocessing or in the case this reprocessing is the actual purpose of the service. The proposed replication solution consists of two types of system generated ids, one that uniquely identifies a unit of data and one that points to the original unit of data representing its source.