AI Model Serving on HPC Infrastructure

We introduce two open-source frameworks for deploying and running large language models (LLMs) within the constraints of high-performance computing (HPC) clusters managed by SLURM job scheduler. One framework is designed for high-throughput batch inference of LLM requests using an OpenAI-compatible format, enabling users to submit traditional HPC-style workloads. Complementing this, the other framework allows dynamically allocating HPC resources to provide interactive serving of LLMs accessible via API calls. Both frameworks enable researchers to request HPC resources for model inference on-demand, eliminating setup complexities and allowing users to focus exclusively on their core research.