Decreasing Time-to-Science with Cloud TPUs on a Campus Slurm Cluster: Roofline, Compiler, and Cost Measurements Across NVIDIA V100, NVIDIA L4, Cloud TPU v5e, and Cloud TPU v6e Trillium

Campus Slurm clusters face long queues on modest NVIDIA GPU nodes while cloud providers hold large Spot Tensor Processing Unit (TPU) pools at lower hourly rates. Datasheet FLOPs do not tell a researcher whether moving CUDA code to OpenXLA finishes a finite batch job sooner, because wall time sums porting, queue wait, node boot, compilation, and step execution. On UC Riverside's Google Cloud Slurm cluster, I built a partition-scoped TPU software stack, static Slurm script checks, and Apptainer container recipes, then benchmarked eight scientific workloads (N = 5 trials, 1 Hz telemetry, CUDA kernel profiling, and OpenXLA HLO cost analysis) across NVIDIA V100 (16 GiB), NVIDIA L4 (24 GiB, 72 W PCIe), Cloud TPU v5e (16 GiB), and Cloud TPU v6e Trillium (32 GiB). Against compiler-matched and concurrency-tuned L4 baselines (torch.compile, XLA-GPU jax.lax.scan, and saturated vLLM batching), TPU v5e and v6e speed up steady-state execution by 2.6x and 2.8x on ColabFold monomer folding (9.1x on isolated AlphaFold2 model inference), 2.1x and 3.6x on vLLM Qwen2.5-3B generation (5.5x at matched 64-sequence concurrency), 2.6x and 5.2x on 336M Transformer training, 3.6x and 4.7x on JAX Lorenz-96 integration, 6.0x and 9.7x on ESM-2 650M embeddings, and 21.0x and 20.5x on XLA-fused Hamiltonian Monte Carlo (392.8x and 383.2x over uncompiled eager PyTorch). At US cloud rates, Spot TPUs deliver 3.0x to 8.1x more steady-state work per dollar than Spot L4 and 7.6x to 20.3x more than on-demand L4. Persistent XLA caches (7.4x to 10.8x faster JAX init; 2.3x faster vLLM init), power-of-2 bucketing, lax.scan loops, and multi-region Spot Apptainer fallback lower wall-clock time-to-science. Author Note and Disclosure: Views and technical findings are the author's own and do not represent the University of California, Riverside or the University of California; campus leadership has not reviewed or approved this text, and nothing here states university procurement or allocation policy. The experimental design, cluster engineering decisions, and conclusions are by Charles E. Forsyth III, who directed AI coding and research agents under written specifications to execute the benchmark suites, check bibliography records against Crossref, run a 20-agent adversarial verification pass, and assist with manuscript drafting from the recorded JSONL, CSV, and NPY telemetry, and reviewed and verified all code, statistical scripts, and final text prior to release.

Authors

Institutions

Publication Details

Journal
Zenodo (CERN European Organization for Nuclear Research)
Published
2026-11-14
DOI
https://doi.org/10.5281/zenodo.24224725
Primary Topic
Parallel Computing and Optimization Techniques
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Decreasing Time-to-Science with Cloud TPUs on a Campus Slurm Cluster: Roofline, Compiler, and Cost Measurements Across NVIDIA V100, NVIDIA L4, Cloud TPU v5e, and Cloud TPU v6e Trillium

Charles E. Forsyth III
Zenodo (CERN European Organization for Nuclear Research)
Parallel Computing and Optimization Techniques
article

Decreasing Time-to-Science with Cloud TPUs on a Campus Slurm Cluster: Roofline, Compiler, and Cost Measurements Across NVIDIA V100, NVIDIA L4, Cloud TPU v5e, and Cloud TPU v6e Trillium

Charles E. Forsyth III
article en

Abstract

Campus Slurm clusters face long queues on modest NVIDIA GPU nodes while cloud providers hold large Spot Tensor Processing Unit (TPU) pools at lower hourly rates. Datasheet FLOPs do not tell a researcher whether moving CUDA code to OpenXLA finishes a finite batch job sooner, because wall time sums porting, queue wait, node boot, compilation, and step execution. On UC Riverside's Google Cloud Slurm cluster, I built a partition-scoped TPU software stack, static Slurm script checks, and Apptainer container recipes, then benchmarked eight scientific workloads (N = 5 trials, 1 Hz telemetry, CUDA kernel profiling, and OpenXLA HLO cost analysis) across NVIDIA V100 (16 GiB), NVIDIA L4 (24 GiB, 72 W PCIe), Cloud TPU v5e (16 GiB), and Cloud TPU v6e Trillium (32 GiB). Against compiler-matched and concurrency-tuned L4 baselines (torch.compile, XLA-GPU jax.lax.scan, and saturated vLLM batching), TPU v5e and v6e speed up steady-state execution by 2.6x and 2.8x on ColabFold monomer folding (9.1x on isolated AlphaFold2 model inference), 2.1x and 3.6x on vLLM Qwen2.5-3B generation (5.5x at matched 64-sequence concurrency), 2.6x and 5.2x on 336M Transformer training, 3.6x and 4.7x on JAX Lorenz-96 integration, 6.0x and 9.7x on ESM-2 650M embeddings, and 21.0x and 20.5x on XLA-fused Hamiltonian Monte Carlo (392.8x and 383.2x over uncompiled eager PyTorch). At US cloud rates, Spot TPUs deliver 3.0x to 8.1x more steady-state work per dollar than Spot L4 and 7.6x to 20.3x more than on-demand L4. Persistent XLA caches (7.4x to 10.8x faster JAX init; 2.3x faster vLLM init), power-of-2 bucketing, lax.scan loops, and multi-region Spot Apptainer fallback lower wall-clock time-to-science. Author Note and Disclosure: Views and technical findings are the author's own and do not represent the University of California, Riverside or the University of California; campus leadership has not reviewed or approved this text, and nothing here states university procurement or allocation policy. The experimental design, cluster engineering decisions, and conclusions are by Charles E. Forsyth III, who directed AI coding and research agents under written specifications to execute the benchmark suites, check bibliography records against Crossref, run a 20-agent adversarial verification pass, and assist with manuscript drafting from the recorded JSONL, CSV, and NPY telemetry, and reviewed and verified all code, statistical scripts, and final text prior to release.

Zenodo (CERN European Organization for Nuclear Research)
University of California, Riverside (US)
Parallel Computing and Optimization Techniques
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon