A YOLO-SSES for Complex Background Document Layout Analysis

In natural scenes, document images often encounter complex background challenges caused by lighting interference, diverse document backgrounds, and variations in shooting angles. These challenges manifest as uneven lighting, shadow occlusion, and other phenomena. Additionally, detection targets vary significantly, including extremely large, extremely small, and occluded objects, posing substantial challenges for Document Layout Analysis (DLA). To address these issues, this paper constructs the Natural Scene Document Layout Analysis Dataset (NSD) to compensate for the limitations of existing datasets in complex background scenarios. Furthermore, an improved layout analysis algorithm based on YOLOv8, named YOLO-SSES, is proposed to enhance detection accuracy in complex scenes. To improve the model’s ability to detect extremely large objects, YOLO-SSES incorporates Shift-ConvNets with large convolutional kernels to refine the C2f structure of YOLOv8. Secondly, to address occlusion in document images, a Separated and Enhancement Attention Module (SEAM) is introduced into the detection head. Through its multi-channel fusion mechanism, SEAM effectively enhances the detection of occluded objects. Additionally, to mitigate the adverse effects of lighting variations such as overexposure and shadows, an Edge Search Module (ES) is designed. Finally, to tackle the challenges associated with detecting extremely small objects, a Shallow Feature Fusion (SFF) module is proposed to efficiently fuse shallow features, further improving detection performance. The proposed algorithm was evaluated on the NSD datasets, achieving mAP scores of 74.7%. These results represent significant improvements of 3.46% compared to the baseline YOLOv8. Code and dataset are available at https://github.com/Capture-U/NDS.