webgguf: browser-native GGUF inference with VRAM-budgeted expert paging for MoE
webgguf runs quantized GGUF language models in a browser tab through WebGPU, without model conversion or server-side inference. Its main mechanism is a fixed-size expert arena: routed MoE weights stay resident when their working set fits the VRAM budget and are paged from OPFS when it does not. This allows a 35B hybrid MoE to run on a laptop GPU with 16 GiB of VRAM while retaining space for the KV cache, recurrent state, activations, and browser allocations. On an RTX 4090 Laptop GPU at a declared 150 W power limit, the best observed warm turn on Qwen3.6-35B-A3B Q2_K reaches 77.0 tok/s. Under the replicated 10-turn protocol, the median steady-state step rate is 71.6 tok/s and the wall-clock generation rate is 60.2 tok/s. Dense decode reaches 224.1 tok/s on Qwen3.5-0.8B Q4_0. In a separate same-host fresh-context comparison on that model, webgguf decodes 2.26 times faster than WebLLM 0.2.84; the TTFT ranges overlap. The evaluation checks individual kernels against an f64 CPU reference, model-level agreement against llama.cpp, shader enablement, and end-to-end performance. The agreement test was also replayed on an Apple M4 Pro; cross-host disagreements occur only at low-margin oracle positions. Conversation checkpoints include both the attention KV cache and the recurrent Gated DeltaNet state. Measurements, runners, and reference outputs are distributed with the implementation.
Authors
- Cristiano Mazzella (ORCID: https://orcid.org/0009-0007-1587-0411)
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-08-26
- DOI
- https://doi.org/10.5281/zenodo.22112595
- Primary Topic
- Parallel Computing and Optimization Techniques
- Type
- preprint