TraCo-CLIP: transition-conditioned CLIP adaptation for static-bias-robust open-vocabulary video action recognition

Abstract Adapting image–text pretrained models to video action recognition requires learning temporal information while preserving transferable visual–textual representations. However, global category-level supervision can favor class-correlated static cues over localized temporal evidence. We propose TraCo-CLIP, a transition-conditioned CLIP adaptation framework for static-bias-robust open-vocabulary video action recognition. A lightweight controller combines semantic change, frame-level ambiguity relative to the source action vocabulary, and their interaction to estimate transition-sensitive routing scores. Temporal non-maximum suppression with fallback selection and confidence-weighted local aggregation construct transition anchors. Two auxiliary objectives supervise the same anchors: transition-centered visual–textual alignment provides inter-class discrimination, while static counterfactual contrast compares transition-oriented and static textual descriptions within the same action category. Both objectives regularize the shared visual representation during training. We evaluate TraCo-CLIP under base-to-novel, cross-dataset, cross-domain open-vocabulary, and out-of-context static-bias protocols. On XOV-Action, the reported controlled comparisons improve $$\mathrm {AVG\text{- }A}$$ by $$2.61$$ and $$2.60$$ percentage points under K150 and K400 source training, respectively. On UCF101-SCUBA/SCUFO, SCUBA accuracy increases by $$8.3$$ points, SCUFO accuracy decreases by $$6.4$$ points, and contrasted accuracy increases by $$11.9$$ points. Controlled ablations and representation-level analyses support the usefulness of the auxiliary learning framework under the evaluated settings. The auxiliary components introduce computation and text-bank storage during training. They are removed for recognition inference, leaving the same deployed architecture and inference operations as the controlled baseline.