FLowRA: Flow-Guided Rank Allocation for Parameter-Efficient Fine-Tuning of Deep Neural Networks
Low-rank adaptation (LoRA) is widely used for parameter-efficient fine-tuning, yet uniform rank assignment ignores the varying contribution of layers to a downstream task. Adaptive methods such as AdaLoRA allocate ranks via gradient and singular-value importance, but they do not exploit the representation change induced by the task. We introduce FLowRA, a calibration-based rank-allocation framework that combines a lightweight flow-change descriptor (velocity, acceleration, and directional alignment of hidden representations) with gradient activity to solve a constrained integer-budget problem. FLowRA complements existing LoRA variants, requires only two user-configurable parameters, and supports multiple architectures through architecture-specific implementations. On GLUE, SuperGLUE, instruction-following, code, vision, and vision-language benchmarks, FLowRA consistently outperforms uniform LoRA, DoRA, and AdaLoRA under low trainable-parameter budgets, reducing the gap to full fine-tuning by 48.5% on GLUE relative to the strongest baseline. We compare FLowRA with AdaLoRA using sensitivity-score correlations and report component ablations to assess the contribution of flow-guided allocation.
Authors
- Valentin Andreevich Malykh (ORCID: https://orcid.org/0000-0002-4508-2527)
- Alexander Savelev (ORCID: https://orcid.org/0009-0007-8348-4731)
- Artem Nazarenko (ORCID: https://orcid.org/0009-0009-8758-9383)
- Elena Mikhailova (ORCID: https://orcid.org/0000-0002-4650-7566)
- Alexandr A. Motyko (ORCID: https://orcid.org/0000-0003-4241-4298)
- Aleksei Toropov (ORCID: https://orcid.org/0009-0008-7555-9446)
- Egor Kotenko (ORCID: https://orcid.org/0009-0007-9835-0584)
- Aleksei Samarin (ORCID: https://orcid.org/0000-0001-6839-3558)
- Aleksandra Dozortseva (ORCID: https://orcid.org/0009-0004-7986-5687)
- Dmitry Nazarenko (ORCID: https://orcid.org/0009-0009-1118-6043)
- Julia Afonina (ORCID: https://orcid.org/0009-0000-7898-8272)
Institutions
- Russian Academy of Sciences (RU)
- International Information Technologies University (KZ)
Publication Details
- Journal
- Machine Learning and Knowledge Extraction
- Published
- 2026-11-19
- DOI
- https://doi.org/10.3390/make8110389
- Primary Topic
- Domain Adaptation and Few-Shot Learning
- Type
- article
- Field-Weighted Citation Impact
- 0.00