AdaptVITS: A Parameter-Efficient Speaker Adaptation Framework for End-to-End Text-to-Speech Under Limited Target Data Constraints

End-to-end voice cloning models like VITS require large amounts of data to produce high-quality speech, making speaker adaptation difficult when target data is limited. To address this problem, this paper introduces AdaptVITS, a parameter-efficient fine-tuning approach evaluated under low-resource multi-speaker joint adaptation constraints, where an aggregate budget of 5 h is distributed across 108 target speakers (∼2.7 min per voice). The strategy freezes the pre-trained text and posterior encoders to keep internal language alignments stable while keeping downstream synthesis modules active. This allows the active 82.50% of parameter capacity to focus entirely on capturing the unique vocal traits of the target speaker. Empirical evaluations across independent 5 h training subsets show that AdaptVITS closely matches the quality of full fine-tuning, achieving a competitive cross-run mean of 3.87 MOS (matching full fine-tuning at 3.84 MOS), 8.59% CER, 15.65% WER, and 0.5188 SECS, while avoiding the alignment failure observed when training from scratch. Furthermore, AdaptVITS reduces mean peak physical GPU memory (VRAM) consumption from 5833.0 MB to 5419.3 MB (a 7.09% reduction). Because this selective optimization occurs strictly during the backward training pass, the final model layout remains unchanged, introducing zero runtime latency or extra storage overhead during deployment. These results position AdaptVITS as a simple, stable, and resource-efficient solution for voice cloning.