Responsive Retrieval of Consistent States in Pipelined Executions of Dataflows

Many modern analytics data workflows (i.e., dataflows) run on distributed data-processing systems to support real-time tasks such as fraud detection and product recommendation. During the execution of a workflow, users often want to inspect two types of states: the internal state of operators and in-flight data tuples between operators (i.e., on the edges) to understand their runtime behaviors for analysis and debugging purposes. While existing methods designed for fault-tolerance can support state retrieval, their response can be slow since they rely on the propagation of a special message (a.k.a. a "barrier" or "marker") on the edges. In this paper, we study how to retrieve a consistent state during a pipelined execution of a workflow with a low latency. We focus on two scenarios: retrieving only operator states, and retrieving both operator and edge states. For each case, we leverage its unique properties to develop a novel retrieval method that does not require barrier propagation on the edges. We also compare these two methods with three existing checkpointing-based methods. We conducted experiments using real datasets and workflows to compare these methods and show that the two novel methods achieve lower latency for state retrieval.