Multi-modal parameter-efficient fine-tuning via hypergraph adapters

Parameter-efficient fine-tuning (PEFT) is an attractive strategy for adapting vision–language models when compute, memory, or labeled data are limited. However, most multimodal PEFT methods still operate on individual samples or pairwise relations, which leaves higher-order structure shared across semantically related examples largely unexploited. We propose HGA-Net, a hypergraph adapter that constructs a mini-batch hypergraph in a joint image–text embedding space and performs lightweight hypergraph message passing inside an adapter bottleneck. To improve stability under noisy captions and ambiguous neighborhoods, we further introduce a soft-incidence formulation that replaces hard hyperedge membership with similarity-weighted participation. We evaluate the method in a scope-matched set-conditioned regime, where inference aggregates only local batch context, and we additionally study a fixed-reference variant for query-independent deployment. Under a unified frozen-backbone interface built on CLIP ViT-B/16 and shared text inputs across all compared modules, HGA-Net achieves top-1 accuracies of 99.71% on Flowers102 and 93.20% on Oxford-IIIT Pets while using 1.573M adapter parameters. The expanded experiments further extend the evaluation to eleven recognition benchmarks, multiple shot budgets (1, 2, 4, 8, and 16 shots), batch-size and batch-composition sensitivity, fixed-reference inference, latency, and a larger CLIP ViT-L/14 backbone. These results indicate that explicitly modeling higher-order cross-sample structure can be an effective inductive bias for set-conditioned multimodal adaptation.