Poster: Coda: Context-aware Acceleration for Distributed LLM Decoding in Edge

The inference of large language models (LLMs) on distributed edge devices is crucial for privacy-preserving applications. However, its performance is severely degraded in a practically lossy edge network due to frequent synchronization. In this paper, we propose Coda, a novel context-aware distributed acceleration framework tailored for packet loss scenarios. We observe the consistency of sparse patterns of neuron group activations in the same context. By identifying sparsity patterns during the prefill phase and updating neuron groups mapping strategically, we enable a relaxed synchronization for decoding, without slowing down, while preserving accuracy. Our approach offers up to 5.62x speedup, significantly outperforming the state-of-the-art in various scenarios.