NICTokenizer: Tokenizer offload to a SmartNIC

Generative AI workloads and services have started to dominate cloud infrastructure capacities with exploding demand in terms of tokens being processed with each training, finetuning or inference workload beginning and ending with Tokenization. Exponentially increasing tokenization volume presents a unique opportunity to leverage network devices already present on the data path for AI pipelines and offload parts of it such as the tokenizer to programmable dataplanes. The tokenizer traditionally run on general purpose CPUs without any hardware accelerators has lightweight hashing as well as table lookups as majority of its tasks. To eliminate host OS kernel overhead and utilize specialized network flow processors we introduce NICTokenizer, a novel framework to offload the tokenizer to SmartNICs to cut down per token latency, boost throughput and freeing up CPU cycles for more intensive workloads. Initial evaluations show that NICTokenizer outperforms the conventional tokenizer implementations run on the server by lowering the latency by 61.3% and increasing the throughput by 407.62% while maintaining a short tail latency guarantee. In addition, offloading the tokenizer to a SmartNIC frees up CPU clock cycles which could be utilized by other processes.