Evaluating Apache Airflow’s Packet Scheduling Latency for API-based data

As the amount of data generated by sensors, mobile phones, and APIs increases day by day, processing static data has become outdated as every data needs to be up to date. Data streaming and processing in real time has become inevitable to analyze and create trends for current events. Apart from continuous data streams, sometimes the data needs to be scheduled for automation tasks. To analyze the efficiency of such tasks, two factors should be considered - the data scheduling latency and the rate of the data being streamed. Data scheduling latency forms the crux of this project as compared to data streaming, which can take a considerable amount of time as it should also process simultaneously. Moreover, the scheduling latency value should be negligible as it is the primary step of a streaming pipeline. Even while the scheduling latency might appear unnecessary, emergency services like healthcare require accurate scheduling which would impact the streaming and synchronization process even if it is affected by a small delay. To calculate the scheduling latency in airflow, a pipeline is implemented with Apache Kafka to stream, Apache Airflow to schedule, Docker to containerize, and an open-source API that acts as a source to provide the data. This pipeline schedules and streams data in real-time followed by finding the scheduling latency of each data packet that is streamed, using its timestamp. Thus, understanding scheduling latency would not only create techniques to minimize it in the long run but also testing, creating benchmarks and comparison of different schedulers is made possible.