SOABC: saliency and object-aware bullet comments in virtual reality videos

Bullet comment, known as "Barrage", is a popular function in Video On Demand (VOD) streaming applications. It refers to time-anchored comments that are directly displayed on top of the video. The comments generally contain the audience’s feedback on the current video scene. Bullet comments are now well-developed for 2D video and applied to famous VOD streaming platforms such as BiliBili in China and Niconico in Japan. Previous study (Li et al. 2022) observed that the 3D spherical display method, compared to the traditional 2D planar display method, is more suitable for bullet comments in 360° videos because of its higher level of immersive experience. However, due to the FoV limitation of VR head-mounted devices, the bullet comments out of the audience’s FoV may be omitted. Our previous work proposed a saliency-aware bullet comment framework for comment insertion to 360° video. It effectively inserts the bullet comments into the audience’s viewing fraction by saliency prediction. However, the study observed that the saliency-aware bullet comments may block the main objects since the high saliency spots are generally the location of the main object. Hence, we propose a dual-branch Saliency and Object-Aware Bullet Comment (SOABC) framework including a saliency map prediction branch and a salient object branch. It allows the bullet comments to be inserted into the high saliency spot while masking the comments if they overlap with the main objects. According to the experimental results obtained, the proposed framework demonstrates the effectiveness of SOABC approach.