Effective Multitasking Segmentation Using Vision Transformer Network for Malformed Images

Background: Effectively and efficiently segmenting human body areas in malformed images is of tremendous practical importance. Recent techniques typically utilize transformers to extract discriminative functions. However, the worldwide interest in these transformers, regular consequences due to the lack of image feature details, and high computational costs result in suboptimal segmentation accuracy and low inference speeds. Objective: To obtain better human segmentation by enhancing model inference speed, this paper applies the Human Spatial Prior Module (HSPM) and the Dynamic Token Pruning Module (DTPM) to deal with real-time three-dimensional environments. Materials and Methods: In this paper, we have applied the Human Spatial Prior Module (HSPM) and the Dynamic Token Pruning Module (DTPM), designed to seize human-unique features in malformed images by dynamically extracting complex information. Meanwhile, the DTPM enhances the inference pace by pruning the point tokens at each Vision Transformer layer (VT). Unlike conventional cropping techniques, the pruned tokens are retained through function maps and selectively reactivated in subsequent layers to enhance model performance. Results: To show the effectiveness of our Vision Transformer for Malformed Images (VT-MI), we expand the dataset with an input image size of 512 x 512 and conduct behavior experiments on this prolonged dataset as well as on the COCO, MPII Human Pose, and a few real-time captured image datasets. Our technique achieves a mIoU growth of 86.5 and an FPS improvement of 7.9 on the COCO dataset with 93.21% accuracy, together with mIoU (person) of 86.11 with dynamic token pruning and reactivation on the entire dataset, while additionally decreasing the model size by using approximately 120 GFLOPs. Conclusion: This work contributes deeper insights into vision transformer networks and provides a clear direction towards both image classification and dense prediction tasks. Moreover, extensive experiments help in showcasing comparative analysis that benchmarks the frameworks contributing to image analysis advancements.