Attention routing in vision-language encoders: technical note
Technical note recording the method and results, as of 14 November 2026, of a study of attention routing in the image encoders of pretrained vision-language models: the attention pattern of these encoders can be decided from a coarse (downsampled or fixed-size thumbnail) view of the image while content is read at full resolution; each query needs a small, fixed budget of keys chosen by the coarse view; and a coarse-routed block-sparse encoder scales near-linearly with resolution (3.9-4.3x faster than the unmodified encoder at about 50k patches on an A100) at matched accuracy on DocVQA, InfoVQA, TextVQA and ChartQA. SHA-256 of the deposited PDF (technical_note.pdf): da07771593364454c4d652fbbb76a213dbacab1876d13975ba2ab0f998985e1c
Authors
- Srinivas Harish
Publication Details
- Journal
- Zenodo (CERN European Organization for Nuclear Research)
- Published
- 2026-11-14
- DOI
- https://doi.org/10.5281/zenodo.24217417
- Primary Topic
- Multimodal Machine Learning Applications
- Type
- article
- Field-Weighted Citation Impact
- 0.00