Shattering the ephemeral storage cost barrier for data-intensive serverless workflows

Serverless computing enables developers to deploy applications as workflows of functions that invoke one another, with cloud providers handling autoscaling and routing. However, serverless platforms lack efficient mechanisms for cross-function data transfers, which hinders the performance of data-intensive applications. Current solutions rely on intermediary services like AWS S3 or ElastiCache(EC), leading to significant cost inefficiencies---storage costs can account for 24-99% of the total execution bill.<br/><br/>Zipline addresses this challenge with a fast, API-compatible data communication method enabling direct function-to-function transfers. Zipline buffers data in the sender function's memory and transmits only the references to the dynamically selected receiver, which pulls the data directly from the sender's memory. While eliminating the need for intermediary services, it also integrates seamlessly with existing autoscaling infrastructure, preserving function invocation semantics while significantly reducing costs and latency. In a vHive/Knative prototype on AWS EC2, Zipline achieves 2-5× lower costs &amp; 1.3-3.4× faster execution times compared to S3. Against EC, Zipline cuts costs by 17-772× while improving performance by 2-5%. Zipline demonstrates a cost-effective and high-performance solution for data-intensive serverless applications.