Federated Inference: Towards Collaborative and Privacy-Preserving Inference over Edge Devices

The growing AI capabilities on edge devices, along with increasing privacy concerns, call for new paradigms for distributed inference. We introduce Federated Inference (FI), a novel framework enabling multiple heterogeneous edge devices to collaboratively execute complex inference tasks on local data without revealing private information. FI pioneers the first design of capability-aware, privacy-preserving model partitioning, where a central orchestra-tor adaptively splits models based on individual client profiles. To ensure privacy, FI injects calibrated differential privacy noise into intermediate activations before transmission. We prototype FI and demonstrate that our adaptive-split strategy significantly reduces latency for weak clients while maintaining privacy and communication efficiency.