Serving Transformer Models via Joint Requst Scheduling and Batching in the Network Edge

Transformers have dominated the field of natural language processing, attributed to their capability to handle sequential input data. There is a surge of work on computational and networking optimizations, aimed at improving the training efficiency of Transformers. However, transformer inference, a cornerstone of myriad AI services, remains relatively underexplored. With the challenge of variable-length inputs, conventional methods adopt padding schemes, resulting in computational waste. Moreover, works on transformer inference often overlook the integration between request scheduling and batching, which play pivotal roles in inference systems. To address these challenges, we introduce TCB, a comprehensive <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">T</b>ransformer inference system that integrates a <bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">C</b>oncat<bold xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">B</b>atching scheme to reduce computational redundancy by concatenating requests. In addition, we present an online request batching algorithm, designed to augment the throughput of scheduled requests. Consider a muiti-server case, we further introduce a joint request assignment and batching scheduling policy to fully utilize resources on servers while ensuring quality-of-service of inference. Extensive experiments demonstrate that our proposed methods can significantly outperform existing works.