Robust real-time detection and tracking of tennis via YOLOBR and FTOC in complex video streams

Detecting tiny, fast-moving objects such as tennis shuttlecocks remains challenging due to their small size, high velocity, and complex backgrounds. This paper demonstrates a hybrid visual system that incorporates classical machine learning and deep learning in intelligent tennis training systems. To address the problem of speed in feature extraction of typical pipelines we propose an AdaBoost based tennis recognition algorithm using HAAR features as well as with integral imaging. A new Frame-difference-based Target Object Compensation (FTOC) algorithm is introduced in order to achieve the real-time tracking and enhance the recalling and robustness of the algorithm by combining three-frame differencing with and temporal trajectory modeling. To reduce computational complexity, the one-stage detector Tiny YOLOv2 is downsized in the deep learning front and introduce two more novel models M-YOLOv2 and YOLOBR. YOLOBR can as well fine-tune the network topology and loss to better capture semantic properties of small flying tennis of various lighting conditions and various background conditions. Based on systematic experiments in a home dataset of tennis videos recorded on 15 video streams, YOLOBR performs better in all three metrics: detection (96.8%), recalls (95.6%), and frame rate (29.3 FPS) than baselines. In addition, FTOC recalls best (94.5) as well as center localization error (CLE = 4.87) in simple and complex scenarios, and it is much better than more sophisticated algorithm, like DSN and COKCF.