ADAPTIVE INCREMENTAL DATA LOADING BASED ON DELTA LAKE IN THE CLOUD ENVIRONMENT

The relevance of the study is due to the growing requirements for the efficient management of large volumes of dynamic data in cloud analytical systems.In the context of the rapid development of lakehouse architectures, there is a need for adaptive models that combine the reliability of transactional mechanisms with high performance of stream processing.The purpose of the study is to form a scientifically based architectural model of incremental data loading in the cloud environment using Delta Lake, which takes into account the specifics of streaming sources, the need for information consistency, and effective detection of data changes.The research methodology is based on the analysis of modern approaches to processing changes in distributed systems, applying the principles of Delta Lake transactional logging, modeling the functional architecture of adaptive loading, and determining performance criteria in parallel processing.Particular attention is paid to the analysis of working with streaming platforms, including Apache Kafka, Amazon Kinesis, and Azure Event Hubs, as well as cloud services such as Databricks, AWS EMR, and Azure Synapse.The results of the study allowed us to identify the key advantages of using Delta Lake in the incremental update mode, and proved the feasibility of dividing the architecture into four functional layers: reception, detection, transformation, and control.The dependence of download efficiency on the quality of the transaction log, scaling parameters, and the system's ability to manage changes in a distributed manner was revealed.The scientific novelty lies in the development of an adaptive architectural model of incremental loading that provides consistent information updates without duplicating processing and allows for a hybrid approach to stream-batch processing.The study's findings confirmed the relevance of using Delta Lake to solve the problems of processing variable data in the cloud environment.It was found that the combination of transaction log mechanisms with streaming processing can increase the performance and flexibility of analytical systems.Prospects for further research include developing tools for predicting conflicts in _delta_log, optimizing resource allocation during peak loads, and formalizing adaptive loading performance metrics in applied scenarios.