End-to-End Coding Based on ViT Feature for Compression Distortion

Standard image codecs are misaligned with the feature space of vision-language models like CLIP, because they are designed to exploit spatial redundancies in pixel grids. In contrast, these features are dominated by channel-wise redundancy and contain only limited spatial structure. We empirically find that ViT-B/32 and ViT-B/16 features exhibit weak yet non-negligible spatial redundancy, sufficient to benefit from light local modeling but not from the aggressive spatial downsampling commonly used in learned image codecs. Guided by this, we develop an end-to-end feature compression framework for ViT-based CLIP. A compact transform maintains essential local structure while focusing modeling capacity on channel dependencies and entropy efficiency. Beyond the codec, we present a distortion-aware post-training strategy that adjusts CLIP to compression-related perturbations. A min–max optimization objective enhances robustness against worst-case feature distortions, and a comparison with an MSE-based post-training variant highlights the benefit of the robust formulation. Experiments on Flickr30k retrieval with \(224\times 224\) inputs show that our method consistently outperforms both traditional codecs (e.g., BPG and VVC intra) and learning based codecs (e.g., MLIC++ and STF), even without post-training. Additional post-training further increases this advantage across bitrates. Additional results with ViT-B/16, latent-space visualizations, kernel-size and stride ablations, and CPU inference measurements support the effectiveness and practicality of the proposed framework.