A vision transformer framework with multi-scale temporal attention for automatic highlight detection and semantic understanding in large-scale sports broadcast video

Automatic highlight detection in sports broadcasting requires models that can localise brief impact events and longer narrative arcs at the same time, and produce meaningful semantic descriptions across sports with markedly different broadcast grammars. Hand-crafted pipelines and single-scale deep architectures struggle on all three fronts, and the difficulty compounds when a system is asked to transfer from one discipline to another. We propose a framework that couples a hierarchical vision Transformer backbone with a three-branch multi-scale temporal attention module and a contrastively aligned semantic decoder, optimised end-to-end under uncertainty-weighted multi-task losses. The backbone equips tube embedding with a learned shot-grammar prior tailored to broadcast footage; the attention module operates at short, medium and long temporal scales fused through a content-dependent gate that adapts to event duration without any explicit duration supervision. A hierarchical decoder emits three semantic axes—action category, participating subjects and event nature—trained jointly with detection through a shared representation. Experiments on SoccerNet-v2, NCAA Basketball and a tennis benchmark report 51.7 ± 0.4 [email protected] and CIDEr 1.42 ± 0.02 over three random seeds, outperforming six competitive baselines including ActionFormer and MViTv2 at p < 0.01. Component-wise ablations isolate the contribution of the shot-grammar prior, the learned duration-aware gate and the shared decoder, and confirm that the gains do not follow from added parameters alone. A qualitative captioning and failure-mode analysis identifies replay overlays, amateur fixed-camera footage and severe occlusion as the three residual weaknesses. Inference runs at 47 ms per 96-frame clip on a single A100 GPU under batch size 1; sustained streaming on a full 90-minute broadcast reaches roughly 2.1× real time once buffering and causal masking are accounted for. Present evidence covers three sports categories only, so we treat the cross-sport claim as encouraging rather than proven; extending it to gymnastics, motorsports and combat sports remains open.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-18
DOI
https://doi.org/10.1038/s41598-026-76229-x
Primary Topic
Video Analysis and Summarization
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

A vision transformer framework with multi-scale temporal attention for automatic highlight detection and semantic understanding in large-scale sports broadcast video

Zhibo Liu, Huaqing Geng
1 links
Scientific Reports
Video Analysis and Summarization
article

A vision transformer framework with multi-scale temporal attention for automatic highlight detection and semantic understanding in large-scale sports broadcast video

Zhibo Liu, Huaqing Geng
article en

Abstract

Automatic highlight detection in sports broadcasting requires models that can localise brief impact events and longer narrative arcs at the same time, and produce meaningful semantic descriptions across sports with markedly different broadcast grammars. Hand-crafted pipelines and single-scale deep architectures struggle on all three fronts, and the difficulty compounds when a system is asked to transfer from one discipline to another. We propose a framework that couples a hierarchical vision Transformer backbone with a three-branch multi-scale temporal attention module and a contrastively aligned semantic decoder, optimised end-to-end under uncertainty-weighted multi-task losses. The backbone equips tube embedding with a learned shot-grammar prior tailored to broadcast footage; the attention module operates at short, medium and long temporal scales fused through a content-dependent gate that adapts to event duration without any explicit duration supervision. A hierarchical decoder emits three semantic axes—action category, participating subjects and event nature—trained jointly with detection through a shared representation. Experiments on SoccerNet-v2, NCAA Basketball and a tennis benchmark report 51.7 ± 0.4 [email protected] and CIDEr 1.42 ± 0.02 over three random seeds, outperforming six competitive baselines including ActionFormer and MViTv2 at p < 0.01. Component-wise ablations isolate the contribution of the shot-grammar prior, the learned duration-aware gate and the shared decoder, and confirm that the gains do not follow from added parameters alone. A qualitative captioning and failure-mode analysis identifies replay overlays, amateur fixed-camera footage and severe occlusion as the three residual weaknesses. Inference runs at 47 ms per 96-frame clip on a single A100 GPU under batch size 1; sustained streaming on a full 90-minute broadcast reaches roughly 2.1× real time once buffering and causal masking are accounted for. Present evidence covers three sports categories only, so we treat the cross-sport claim as encouraging rather than proven; extending it to gymnastics, motorsports and combat sports remains open.

Scientific Reports
Dongshin University (KR), China Academy of Chinese Medical Sciences (CN)
Video Analysis and Summarization
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon