Machine learning based convex hull prediction for content-aware ABR encoding

After it was initially published by Netflix, convex hull based dynamic optimizer framework has been widely used in the industry for Adaptive Bit Rate (ABR) streaming. Improved methods, such as a two-stage convex hull based dynamic optimizer, provide the better compression efficiency based on video characteristics. This approach requires a very large amount of pre-encoding operations in the first stage, to predict the convex hull. Due to its complexity, it is only enabled for highly watched videos and important use cases. In order to reduce its complexity, different approaches have been studied. In this paper, we summarize the high-level requirement of the system and describe a few options that have been explored, including a machine-learning based method that is used to predict a complete set of encoding statistics, including bitrate and quality, from the result of only a few pre-encodings and other spatial and temporal complexity metrics of the video. Based on the predicted encoding statistics, the same convex hull dynamic optimizer algorithm can be used to derive the optimal encoding configurations (resolution and bitrate), with only a few first stage encoding operations. With this approach, a significant amount of compute cost can be reduced without sacrificing compression efficiency. Therefore, we can enable content-aware encoding production to more videos at scale. In this paper, we present the details of these few approaches, their experiment results, and directions for some future works.