Measured Energy per Token for Four Open Language Models on NVIDIA H100 and B200: Batch, Context, Precision, Tensor Parallelism and Prefix Caching

How much energy does a GPU use to produce one token of a language model’s answer, and what settings move that number? We metered the board energy of rented NVIDIA H100 and B200 GPUs serving four open-weight models (Qwen3-8B, Qwen3-30B-A3B, DeepSeek-V2-Lite and Llama-3.3-70B) with vLLM, in 917 runs over 472 configurations, mainly on two rented pods per GPU type (13 pods in 8 data centers in all). Every figure is gross board energy, idle draw included, over a defined window of requests. Our research shows that batch concurrency was the largest lever: between one request at a time and the highest load we offered (67 to 91 requests running on average), energy per output token fell 14- to 42-fold (median 20-fold). Latency increased with batch depth: the median time to first token rose from 0.03 to 0.05 s at one request to 0.2 to 1.0 s at a target of 64. Reusing a cached prompt came next: a request whose first 16k tokens were already in the server’s cache used 1.5 to 6 times less energy than an uncached request of the same length. Running past the server’s memory for stored context raised energy per token 4- to 6-fold, and a longer context raised it 1.3 to 2.7 times at a target batch of 8. Splitting a model over additional GPUs cost 7 to 40% more energy per token for 1.1 to 1.6 times the throughput, and 4-bit (nvfp4) versions of two of the models, as published by Red Hat and NVIDIA, used 14 to 39% less energy per output token than the 8-bit versions on B200 (one pod per model). At the same target batch and on the same number of GPUs, the B200 gave no consistent saving over the H100 (a median 0.93 of its energy per token, from 0.75 to 2.7 depending on the model), a comparison that leaves the faster GPU less fully used: at one request at a time neither GPU reached two-thirds of its memory-bandwidth bound, and the B200 was further from it than the H100 for every model. Serving Llama-3.3-70B on one B200 instead of the two H100s it needed used 7 to 30% less energy per token at batches 1 to 64, partly from using fewer GPUs. A loaded, idle GPU drew about a fifth (H100) to a quarter (B200) of its rated power. The corpus, harness and analysis script are released with the paper. This record holds the paper (MeasuredEnergyperTokenFourOpenLanguageModels.pdf) and everything needed to check it: corpus.tar.gz (features.parquet, one row per run, and store_tables/cells.parquet, one row per window); raw.tar.gz (per-run records, request records, raw power samples, counter qualification and server logs for all 917 runs); code.tar.gz (the repository at tag v1.0.0: paper source, analysis script, the metering harness as it ran, and tools); and SHA256SUMS covering every file. Running analysis/build.py on the corpus regenerates every table and figure. Licensing: paper text CC BY-NC-ND 4.0; data CC BY-NC 4.0 with express permissions for regulators, standards bodies, public authorities and auditors; code Apache-2.0.