Memory-efficient Model Growth for Finetuning Automatic Speech Recognition Models

Scaling up neural networks improves performance but increases memory requirements, making efficiency a key challenge in modern AI training. While pruning, quantization, and parameter-efficient finetuning (PEFT) reduce memory constraints, they do not directly enhance model capacity. Model growth addresses this by expanding a trained base model, typically through width-wise or depth-wise expansion, but continued pretraining remains costly. We propose a memory-efficient depthwise model growth approach for finetuning automatic speech recognition (ASR) models. By duplicating transformer layers and applying LoRA, we enhance model capacity while minimizing memory overhead. The duplicated layers share weights, further reducing memory demands. Our experiments on Dutch ASR demonstrate a 10% relative improvement over the baseline with the same parameter budget. This work offers a novel method for scaling ASR models efficiently without the computational burden of continued pretraining.