Poster: Optimizing Transmission for Privacy-Preserving Edge-Cloud Split LLM Inference

Today’s cloud-centric Large Language Model (LLM) inference can raise privacy concerns, as raw user data is sent over public networks. Split LLM inference offers a promising alternative by handling sensitive stages, such as prefill and input/output decoding, locally at the edge while offloading heavy intermediate decoding layers to the cloud. This approach could enhance privacy, yet it introduces a key challenge: how to transfer large intermediate vectors with ultra-low latency and high reliability over dynamic networks. We explore a potential transmission framework combining (i) dynamic congestion control to adapt flow and reduce queueing/loss, and (ii) adaptive FEC and quantization to balance redundancy with compression. Preliminary results indicate that this approach can achieve lower per-token latency and higher throughput than TCP/QUIC, enabling privacy-preserving LLM services that remain highly responsive and aligned with real-time user experience demands.