A Polynomial Coded Computing Scheme for Parallel Computation of Multi-Head Attention

Transformer-based models, based on multi-head attention, have revolutionized the field of deep learning and have excelled in a wide range of tasks. The core computation task, multi-head attention, is typically carried out in distributed computing framework, where stragglers are the most important factor that affects inference efficiency. To this end, we propose to alleviate the effect of stragglers by coding techniques. In particular, polynomial coding is applied to the parallel computation of multi-head attention block, which achieves the optimal tradeoff between computation load and robustness against stragglers.