TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers save computation but sacrifice quality. We propose Test-Time Baking (TTB), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks. Project page: https://test-time-baking.github.io
Authors
- Ryo Hachiuma (ORCID: https://orcid.org/0000-0001-8274-3710)
- Wei-Chen Chiu (ORCID: https://orcid.org/0000-0001-7715-8306)
- Jaesung Choe (ORCID: https://orcid.org/0000-0002-8978-6702)
- Min-Hung Chen
- Cheng Sun
- Yu-Chiang Frank Wang
- Chin-Yang Lin
- Yu-Lun Liu
- Yen-Yu Lin
Publication Details
- Journal
- arXiv (Cornell University)
- Published
- 2026-11-12
- DOI
- https://doi.org/10.48550/arxiv.2611.17331
- Primary Topic
- 3D Shape Modeling and Analysis
- Type
- preprint