Multi‐Head Bias Fusion and Adaptive Aligned Gradient of Multi‐Task Learning for Joint Human Detection, Segmentation and Pose Estimation

ABSTRACT Human object detection, segmentation and pose estimation, are critical issues in computer vision research. The simultaneous application of multiple independent networks for these tasks significantly increases the computational burden. To achieve efficiency while maintaining high accuracy, we propose a novel multitask learning network: You Only Look Once for object detection, instance segmentation and pose estimation (YOLODSP). YOLODSP incorporates multi‐task heads and introduces a BiasFusion module to predict the offsets of pose estimation and segmentation. Furthermore, an adaptive multitask loss function that incorporates the label mask and imbalanced weight into the keypoint loss function is designed to effectively mitigate the impact of inconsistent label quantities. In addition, to reduce the conflict of multiple tasks on the shared encoder, an adaptive aligned gradient method is proposed to adjust the gradients of each task, enhancing the adaptability of the encoder to multiple tasks. The results show that YOLODSP outperforms the existing methods on challenging benchmarks such as COCOPersonAll and OCHuman. With an overall difference in average precision of no more than 1% relative to the combination of multiple single‐task methods, YOLODSP reduces the computational load by 35.2%, 38.8% and 40.5% on the nano, medium and x‐large backbones, respectively.