Invited: SambaNova SN40L: Unleashing Agentic AI with Dataflow

AI inference clouds are increasingly being tasked with running a diverse set of models to support interactive, agentic workloads such as LLM-powered assistants, chatbots, and autonomous agents. Efficient AI cloud inference requires producing tokens during the memory-bound decode phase at peak performance, as well as economically hosting and rapidly switching between a vast array of models. We describe the SambaNova SN40L Reconfigurable Dataflow Unit (RDU) that combines dataflow with a three-tier memory system with SRAM, HBM, and DDR. Dataflow enables peak token generation performance with aggressive fusion of large compute graphs into a single kernel. HBM and high-capacity DDR drastically lower hardware footprint to host and serve trillions of parameters at scale. The SN40L RDU produce tokens over $\mathbf{3} \times$ faster, consume $3 \times$ less energy, and lower model hosting costs by up to $19 \times$ over a DGX H100.