Efficient Serverless Stream Processing based on High-level Programming and Parallelism AutoTunig

Stream processing jobs often require elastic scaling in response to dynamically changing loads. In recent years, the development of serverless technology has provided new potential for elastic scaling of stream processing jobs. However, directly implementing stream processing jobs to serverless platforms faces many challenges such as program complexity, high latency and load imbalance. In this paper, we propose an efficient stream processing system under serverless environment. Our system provides a high-level programming interface and can automatically determine the parallelism under the current load through a non-linear model. Additionally, it reduces the latency and increases the throughput of stream processing jobs through decentralized orchestration and soft affinity scheduling policies. We conducted a series of experiments to evaluate the performance of our system. The experimental results show that our system effectively improves the throughput and reduces the latency of stream processing jobs under serverless environment compared with the widely-used scheduling and orchestration methods used in serverless environment.