Ternary KV Cache for a 27B Model on a Phone, Preliminary Functional Report

Preliminary technical note on a ternary KV cache for a 27B parameter Qwen3.5-class model. The cache stores both K and V in TQ1_0 or TQ2_0 format and runs through the non-flash attention path with Q8_K activations. The model is Ternary Bonsai 2 27B with 64 blocks, 16 full attention layers, 48 SSM layers, and head_dim 256. Functional correctness was tested on a Windows PC and on one Android phone. Both cache types produced coherent text. On the phone the TQ2_0 cache reached 6.14 to 6.37 tok/s prompt processing and 3.19 to 3.27 tok/s generation on a 32 token prompt and 8 generated tokens. The TQ1_0 cache was about 7 percent slower than q8_0 on decode and saved memory. The phone reached 67 to 68 C in about 30 seconds. The note records what is confirmed and what is not. Sustained performance, a real 32k context run, the MIUI application path, other SoCs, and GPU support remain open. No priority claim is made. Contents include the note, the TQ1_0 and TQ2_0 2x2 ARM kernel patches, the llama.rn ternary KV patch, a SHA256 manifest, and an OpenTimestamps proof. Text license CC BY 4.0. Code license MIT. Public handle figkcoci. Version 0.1.1 replaces the initial pending OpenTimestamps proofs with upgraded proofs. The note, manifest and archive are unchanged from 0.1.0. The upgraded proofs contain BitcoinBlockHeaderAttestation(974483). The calendars committed the proofs to Bitcoin transactions 6793ee832ed4620edb3ec75abefe6400564398149817d882cbc63962cb6e4040 and f098822a94cde8d206f411d8286ea769c9638d95d4b142e214ea56865b3f7282 in block 974483, whose merkle root is bd0ad171564ea008cc9206e6fb0929afac7e9e7b6dda84b3a41d2cad0942fdac. The record also includes BITCOIN_PROOF.txt with the block hash, merkle root, transaction ids and verification commands. Version 0.1.2 corrects an indexing error in the TQ1_0 2x2 ARM kernel that was present in 0.1.1. The decoder used q[j*5+k] instead of q[k*2+j]. The error affected even-length prompt prefill. Single-token decode uses the 1-row path and was not affected. The corrected patch and ERRATUM_2026-11-04.md are included. The 17.9 percent decode gain is withdrawn. The 44.5 percent prefill gain was measured with the incorrect kernel and is pending re-measurement. TQ2_0 2x2 is unaffected. The erratum, manifest and archive proofs are in Bitcoin block 974522. The unchanged note keeps its 974483 proof.

Authors

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-04
DOI
https://doi.org/10.5281/zenodo.23937993
Primary Topic
Advanced Neural Network Applications
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Ternary KV Cache for a 27B Model on a Phone, Preliminary Functional Report

Egor Pobozhii
Zenodo (CERN European Organization for Nuclear Research)
Advanced Neural Network Applications
article

Ternary KV Cache for a 27B Model on a Phone, Preliminary Functional Report

Egor Pobozhii
article en

Abstract

Preliminary technical note on a ternary KV cache for a 27B parameter Qwen3.5-class model. The cache stores both K and V in TQ1_0 or TQ2_0 format and runs through the non-flash attention path with Q8_K activations. The model is Ternary Bonsai 2 27B with 64 blocks, 16 full attention layers, 48 SSM layers, and head_dim 256. Functional correctness was tested on a Windows PC and on one Android phone. Both cache types produced coherent text. On the phone the TQ2_0 cache reached 6.14 to 6.37 tok/s prompt processing and 3.19 to 3.27 tok/s generation on a 32 token prompt and 8 generated tokens. The TQ1_0 cache was about 7 percent slower than q8_0 on decode and saved memory. The phone reached 67 to 68 C in about 30 seconds. The note records what is confirmed and what is not. Sustained performance, a real 32k context run, the MIUI application path, other SoCs, and GPU support remain open. No priority claim is made. Contents include the note, the TQ1_0 and TQ2_0 2x2 ARM kernel patches, the llama.rn ternary KV patch, a SHA256 manifest, and an OpenTimestamps proof. Text license CC BY 4.0. Code license MIT. Public handle figkcoci. Version 0.1.1 replaces the initial pending OpenTimestamps proofs with upgraded proofs. The note, manifest and archive are unchanged from 0.1.0. The upgraded proofs contain BitcoinBlockHeaderAttestation(974483). The calendars committed the proofs to Bitcoin transactions 6793ee832ed4620edb3ec75abefe6400564398149817d882cbc63962cb6e4040 and f098822a94cde8d206f411d8286ea769c9638d95d4b142e214ea56865b3f7282 in block 974483, whose merkle root is bd0ad171564ea008cc9206e6fb0929afac7e9e7b6dda84b3a41d2cad0942fdac. The record also includes BITCOIN_PROOF.txt with the block hash, merkle root, transaction ids and verification commands. Version 0.1.2 corrects an indexing error in the TQ1_0 2x2 ARM kernel that was present in 0.1.1. The decoder used q[j*5+k] instead of q[k*2+j]. The error affected even-length prompt prefill. Single-token decode uses the 1-row path and was not affected. The corrected patch and ERRATUM_2026-11-04.md are included. The 17.9 percent decode gain is withdrawn. The 44.5 percent prefill gain was measured with the incorrect kernel and is pending re-measurement. TQ2_0 2x2 is unaffected. The erratum, manifest and archive proofs are in Bitcoin block 974522. The unchanged note keeps its 974483 proof.

Zenodo (CERN European Organization for Nuclear Research)
Openalex Percentile: Top 19%
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon