Decentraliserad felåterställning i distribuerade stream behandlingssystem med CRDTs

Distributed stream processing systems are central to modern real-time data analytics but face critical challenges in terms of fault tolerance. Traditional systems rely on centralized failure recovery that require global coordination, which results in checkpointing overhead and system-wide rollbacks in the event of failures. These limitations are especially pronounced in global aggregation workloads, where events of the same key are often shuffled to a single node for processing, creating a bottleneck for normal operation, checkpointing and recovery. This thesis explores a novel decentralized failure recovery approach for stateful stream processing systems, performing global aggregation by leveraging Conflict-Free Replicated Data Types (CRDTs). Using CRDTs, we provide strong eventual consistency and enable decentralized state management without global coordination. Based on CRDTs, we design and implement a decentralized failure recovery module that provides exactly-once processing semantics while allowing local and parallel recovery of failed nodes. Thus reducing coordination overhead and eliminating the need for global rollbacks. Moreover, it integrates work stealing for dynamic load balancing to improve operation under variable workloads. We evaluate our system against Apache Flink under normal operation and different failure conditions using the Nexmark benchmark suite. The results showcase reduced checkpointing overhead and significant improvements in recovery time compared to Flink, by demonstrating sub-2-second recoveries across all failure scenarios. Overall, our system demonstrates greater resilience to failures than Flink. These findings suggest that CRDTs can significantly enhance the performance and robustness of stateful, decentralized stream processing systems.