Hayula Research Paper

Large language model (LLM) inference on consumer-grade hardware remains bottlenecked by autoregressive decoding, where each token requires a separate forward pass through the model. Speculative decoding mitigates this by using a small draft model to propose multiple tokens per step, but introduces a verifier overhead that can negate gains on memory-bandwidth-constrained systems such as Apple Silicon. We present PIPO-Hayula, an implementation of the Pair-In-Pair-Out (PIPO) multi-token prediction framework (arXiv:2605.27255) adapted for the Apple M2 Ultra system-on-a-chip. PIPO replaces the traditional draft-verify pipeline with a single latent-space transformation that predicts two tokens from two input tokens in a single forward pass, eliminating the verifier entirely. Our implementation integrates with Ollama and llama.cpp to target Qwen2.5:72b (Q4\_K\_M, 47.4 GB) running at 10–15 tok/s on a Mac Studio M2 Ultra with 192 GB unified memory. We demonstrate a theoretical first-token speedup of up to 2.64× over standard autoregressive decoding, with substantially lower memory overhead than conventional speculative decoding approaches. The PIPO-Hayula module is deployed as part of the Hayula Gateway fallback chain, routing through local M2 models → Qwen2.5:72b → OpenRouter, serving approximately 199,000 requests per month. Our results establish PIPO as a practical, production-ready technique for accelerating local LLM inference on memory-bandwidth-limited Apple Silicon hardware.