Codesign of Tensors Encoding And Transcoding: A Building Block For Decentralized AI

Decentralized AI systems must transmit data/model updates to clients operating under diverse bandwidth, compute, and memory constraints. Traditional approaches either downcast tensors to a fixed low-precision format-sacrificing accuracy-or transmit full-precision values, overwhelming resource-constrained clients and increasing training time due to stragglers. Moreover, sending different precisions to different clients rule out the possibility of using (switch based or overlay) broadcast/multicast primitives as typically, two different precisions are essentially two different bytestreams. Inability to use broadcast primitives leads to high resource utilization in an resource constraint environment. We introduce xFP, a tensor encoding scheme that enables decoding multiple precisions from the same encoded data based on how much data is decoded by a client (Figure 1). So the same encoded tensors can be broadcasted to all clients where clients decode as much precision as they can support based on how big a prefix of packets they consume. Github repository: https://github.com/HaseebLUMS/adaptive-fp-encoding