A Lightweight Model for Fast and Detail-Preserving Human Segmentation in Video

With the rapid development of artificial intelligence, human segmentation in video is becoming increasingly important in the field of computer vision. However, existing segmentation models suffer from inaccurate segmentation, slow processing, and large model size that limit their deployment on resource-constrained devices. To this end, we propose a lightweight model called Efficient Memory Aggregation U-shaped Network (EMAUnet) for human segmentation in video, which is based on a traditional U-shaped network and attention mechanism. In EMAUnet, memory modules are combined with segmentation modules, enabling end-to-end learning of semantic extraction patterns for human images. Mobile-inverse bottleneck convolution (MBConv) is used as the network backbone that has relatively few parameters and computational complexity. Inverted sub-pixel down-sampling (ISP) is proposed to minimize information loss and achieve detail-preserving of segmentation. Coordinate attention (CA) is adopted to precisely locate the portrait area. Moreover, bidirectional memory update (BMU) and memory update trigger (MUT) are proposed to improve memory resource utilization and reduce unnecessary computation. Experimental results show that, compared with the classic model ISNet, EMAUnet has the mIoU, FPS and pixel accuracy increased by 2.3%, 25.0% and 1.8%, respectively, while the amount of parameters and the size of model decreased by 33.3% and 45.7%, respectively.