thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware

Small language models on edge hardware spend their decode step in launch overhead and memory traffic, not arithmetic. thox-edge-ops v2 is a set of six fused inference operators (RMSNorm with a weight offset that covers the Llama and Gemma conventions, fused residual-add RMSNorm, SiLU and tanh-GELU gated activations, rotary embedding, and a fused per-head QK-norm plus RoPE prologue) with one source tree and three backends: CUDA, Metal and CPU, distributed through the Hugging Face Kernel Hub as 23 ABI-checked x86_64 builds, with the full 438-test suite passing against 3 builds downloaded from the Hub. At decode shapes the GPU kernels run 6.90x faster than eager PyTorch for RMSNorm and 24.43x faster for the fused QK-norm and RoPE prologue on an NVIDIA RTX 4060 Ti; on the CPU, RMSNorm runs 2.42x and rotary embedding 6.75x faster. The revised public edition adds a deployment section covering the THOX stack (edge hardware, ThoxOS and ThoxInux, MeshStack, the kernels, the small models) and a hardware qualification matrix: the x86_64 workstation and hub class is MEASURED for v2; Jetson Orin, Apple silicon and the Arm device class (ThoxAir, ThoxMini, ThoxClip) are targets only, with no aarch64 Hub build published for v2 and no Arm measurement claimed. Preprint, not peer reviewed. THOX.ai Research.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-05
DOI
https://doi.org/10.5281/zenodo.23971226
Primary Topic
Parallel Computing and Optimization Techniques
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware

Tommy Xaypanya
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
preprint

thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware

Tommy Xaypanya
preprint en

Abstract

Small language models on edge hardware spend their decode step in launch overhead and memory traffic, not arithmetic. thox-edge-ops v2 is a set of six fused inference operators (RMSNorm with a weight offset that covers the Llama and Gemma conventions, fused residual-add RMSNorm, SiLU and tanh-GELU gated activations, rotary embedding, and a fused per-head QK-norm plus RoPE prologue) with one source tree and three backends: CUDA, Metal and CPU, distributed through the Hugging Face Kernel Hub as 23 ABI-checked x86_64 builds, with the full 438-test suite passing against 3 builds downloaded from the Hub. At decode shapes the GPU kernels run 6.90x faster than eager PyTorch for RMSNorm and 24.43x faster for the fused QK-norm and RoPE prologue on an NVIDIA RTX 4060 Ti; on the CPU, RMSNorm runs 2.42x and rotary embedding 6.75x faster. The revised public edition adds a deployment section covering the THOX stack (edge hardware, ThoxOS and ThoxInux, MeshStack, the kernels, the small models) and a hardware qualification matrix: the x86_64 workstation and hub class is MEASURED for v2; Jetson Orin, Apple silicon and the Arm device class (ThoxAir, ThoxMini, ThoxClip) are targets only, with no aarch64 Hub build published for v2 and no Arm measurement claimed. Preprint, not peer reviewed. THOX.ai Research.

Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon
thox-edge-ops v2: fused decode kernels for small models on THOX edge hardware — Tommy Xaypanya · Zenodo (CERN European Organization for Nuclear Research) (2026) | TGRS Research Map | TGRS