webgguf: browser-native GGUF inference with VRAM-budgeted expert paging for MoE

webgguf runs quantized GGUF language models in a browser tab through WebGPU, without model conversion or server-side inference. Its main mechanism is a fixed-size expert arena: routed MoE weights stay resident when their working set fits the VRAM budget and are paged from OPFS when it does not. This allows a 35B hybrid MoE to run on a laptop GPU with 16 GiB of VRAM while retaining space for the KV cache, recurrent state, activations, and browser allocations. On an RTX 4090 Laptop GPU at a declared 150 W power limit, the best observed warm turn on Qwen3.6-35B-A3B Q2_K reaches 77.0 tok/s. Under the replicated 10-turn protocol, the median steady-state step rate is 71.6 tok/s and the wall-clock generation rate is 60.2 tok/s. Dense decode reaches 224.1 tok/s on Qwen3.5-0.8B Q4_0. In a separate same-host fresh-context comparison on that model, webgguf decodes 2.26 times faster than WebLLM 0.2.84; the TTFT ranges overlap. The evaluation checks individual kernels against an f64 CPU reference, model-level agreement against llama.cpp, shader enablement, and end-to-end performance. The agreement test was also replayed on an Apple M4 Pro; cross-host disagreements occur only at low-margin oracle positions. Conversation checkpoints include both the attention KV cache and the recurrent Gated DeltaNet state. Measurements, runners, and reference outputs are distributed with the implementation.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-08-26
DOI
https://doi.org/10.5281/zenodo.22112595
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
preprint

webgguf: browser-native GGUF inference with VRAM-budgeted expert paging for MoE

Cristiano Mazzella
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

webgguf: browser-native GGUF inference with VRAM-budgeted expert paging for MoE

Cristiano Mazzella
preprint en

Abstract

webgguf runs quantized GGUF language models in a browser tab through WebGPU, without model conversion or server-side inference. Its main mechanism is a fixed-size expert arena: routed MoE weights stay resident when their working set fits the VRAM budget and are paged from OPFS when it does not. This allows a 35B hybrid MoE to run on a laptop GPU with 16 GiB of VRAM while retaining space for the KV cache, recurrent state, activations, and browser allocations. On an RTX 4090 Laptop GPU at a declared 150 W power limit, the best observed warm turn on Qwen3.6-35B-A3B Q2_K reaches 77.0 tok/s. Under the replicated 10-turn protocol, the median steady-state step rate is 71.6 tok/s and the wall-clock generation rate is 60.2 tok/s. Dense decode reaches 224.1 tok/s on Qwen3.5-0.8B Q4_0. In a separate same-host fresh-context comparison on that model, webgguf decodes 2.26 times faster than WebLLM 0.2.84; the TTFT ranges overlap. The evaluation checks individual kernels against an f64 CPU reference, model-level agreement against llama.cpp, shader enablement, and end-to-end performance. The agreement test was also replayed on an Apple M4 Pro; cross-host disagreements occur only at low-margin oracle positions. Conversation checkpoints include both the attention KV cache and the recurrent Gated DeltaNet state. Measurements, runners, and reference outputs are distributed with the implementation.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

webgguf: browser-native GGUF inference with VRAM-budgeted expert paging for MoE — Cristiano Mazzella · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS