Subscription-Free AI: Energy-Efficient On-Device Inference of Quantized Language Models on Low-Cost Edge Hardware

Most people use AI by renting it from a cloud service, paying per token or per month and sending their data off the device. We ask whether low-cost local hardware can do the job instead, using only free data and open tools. Study A analyses 243 de-duplicated MLPerf Tiny results (v1.0 to v1.3) measured on microcontrollers, neural processors and FPGAs. Int8 made Cortex-M inference 1.9x to 3.8x faster than float32 with no loss in classification accuracy, a dedicated NPU spotted keywords 25x faster than the median Cortex M4, and energy per inference tracked latency with a log-log slope of 0.97 (R² = 0.72). Study B trains three transformer language models (0.8M to 6.4M parameters) on a 2-core CPU with a dependency-free NumPy implementation and evaluates post-training quantization from 16 to 1.58 bits. Four-bit weights in blocks of 32 raised validation loss by only 0.09% to 0.55% while shrinking the models 6.4x to 6.9x; below 4 bits the loss depended on block size, model scale and layer type, and clipping-based scale search reduced weight error yet increased model error. Portable C kernels decoded 0.4B- and 1.1B-scale weight geometries 2.7x faster at 4 bits than at FP32. Roofline, energy and cost models translate these results into bandwidth requirements, energy per token and the volume at which owning hardware beats renting a service.

Authors

Publication Details

Journal
Preprints.org
Published
2026-11-12
DOI
https://doi.org/10.20944/preprints202611.1268.v1
Primary Topic
Advanced Neural Network Applications
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Subscription-Free AI: Energy-Efficient On-Device Inference of Quantized Language Models on Low-Cost Edge Hardware

Subramaniam Ganesan, Ajay A. Waghmare
Preprints.org
Advanced Neural Network Applications
preprint

Subscription-Free AI: Energy-Efficient On-Device Inference of Quantized Language Models on Low-Cost Edge Hardware

Subramaniam Ganesan, Ajay A. Waghmare
preprint en

Abstract

Most people use AI by renting it from a cloud service, paying per token or per month and sending their data off the device. We ask whether low-cost local hardware can do the job instead, using only free data and open tools. Study A analyses 243 de-duplicated MLPerf Tiny results (v1.0 to v1.3) measured on microcontrollers, neural processors and FPGAs. Int8 made Cortex-M inference 1.9x to 3.8x faster than float32 with no loss in classification accuracy, a dedicated NPU spotted keywords 25x faster than the median Cortex M4, and energy per inference tracked latency with a log-log slope of 0.97 (R² = 0.72). Study B trains three transformer language models (0.8M to 6.4M parameters) on a 2-core CPU with a dependency-free NumPy implementation and evaluates post-training quantization from 16 to 1.58 bits. Four-bit weights in blocks of 32 raised validation loss by only 0.09% to 0.55% while shrinking the models 6.4x to 6.9x; below 4 bits the loss depended on block size, model scale and layer type, and clipping-based scale search reduced weight error yet increased model error. Portable C kernels decoded 0.4B- and 1.1B-scale weight geometries 2.7x faster at 4 bits than at FP32. Roofline, energy and cost models translate these results into bandwidth requirements, energy per token and the volume at which owning hardware beats renting a service.

Preprints.org
Advanced Neural Network Applications
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon