A multi-operator deep-learning approach to preserve visual integrity in content-aware image retargeting task

Abstract Preserving key visual elements during content-aware image retargeting is a challenging task as the integrity of prominent visual features are often destroyed while resizing images. Seam carving (SC) also known as content-aware image retargeting is a technique that adds or removes paths of pixels with the least importance to adjust the images dimensions while preserving important visual elements. Although Seam carving has proven to be an effective tool to resize images that removes low-importance seams, it often fails to preserve the overall structure of the image that result in distortions and deformations in retargeted images under extreme aspect ratio adjustments. The aim of this study is to enhance content-aware image retargeting results by developing a multi-operator pipeline to improve preservation of key visual elements during resizing and ultimately overcome the shortcomings of traditional seam carving in preserving visual integrity of retargeted images with complex compositions. The proposed multi-operator technique integrates Mask R-CNN into the seam carving process to automate common object detection and object masking of retargeted images, where scaling serves as a fallback mechanism to achieve the required dimensions. The proposed MODL-CAIR approach integrates automatic object masking into the image retargeting pipeline to generate a constraint-aware energy map while introducing a seam banning mechanism for semantic-guided seam removal. The results of this study demonstrate that protecting salient objects can effectively preserve prominent visual regions and overall structure of the image during the image retargeting process.