CustomFit: Customizing Task-Optimized and Size-Specific Subnets from a Single Base Network During Runtime

We address the challenge of efficiently deploying networks across hardware platforms with varying resource constraints and task requirements. While conventional methods train a base network that can be split into subnets of different sizes for deployment, they typically focus on a single task. This paper introduces CustomFit, a framework that trains a base network from which different-sized subnets tailored to multiple tasks can be segmented during inference. CustomFit uses task-specific feature mapping to optimize the base network for each task without altering its weights. Subnet segmentation is guided by the task’s feature mapping process and the desired size. By training a base network with CustomFit on Speech Enhancement (SE) and Spoken Language Understanding (SLU) tasks, we show that 30 subnets of varying sizes can be segmented. Compared to state-of-the-art dynamic networks, CustomFit achieves similar SE scores and improves SLU accuracy by 2.75%, while using 2.9× fewer parameters in the base network. The parameter sizes of task-specific subnets range from 11k to 720k, enabling flexible deployment across a range of edge platforms from microcontroller units to mobile phones.