CF-STAR: Highly compressible adapters for model merging via centralized task vectors

Abstract Model merging offers a training-free solution to the prohibitive storage costs of the one-model-per-task paradigm. However, current merging techniques present a difficult trade-off. Creating a single model is storage-efficient but hurts performance, while adapter-based methods perform well but introduce a new storage burden. This work introduces CF-STAR, a framework that resolves this dilemma by creating highly compressible adapters. Our approach redefines the adapter itself. Instead of representing the deviation from a pre-trained model, our adapter is defined by the deviation from the multi-task average. We call this new representation the centralized task vector (CTV). This CTV represents a purer form of task-specific knowledge within the merging context, making it fundamentally more compressible. CF-STAR exploits this with a novel low-rank plus sparse decomposition tailored to this representation, capturing both global structure and critical details. Furthermore, the entire pipeline is designed to be synergistic with low-bit quantization, further enabling extreme compression. On diverse benchmarks spanning image classification, NLP, and dense prediction, CF-STAR sets a new state of the art on the accuracy-storage Pareto frontier, achieving up to $$40\times $$ <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"> <mml:mrow> <mml:mn>40</mml:mn> <mml:mo>×</mml:mo> </mml:mrow> </mml:math> adapter compression over strong baselines while maintaining competitive performance.