NAST: neural architecture search and transformer for image manipulation localization

Image manipulation localization (IML) is one of the hot topics in the community of media forensics. Deep learning approaches have shown their great power for IML in recent years. However, most of the approaches rely severely on network structures manually designed by experts. At the same time, the transformer has exhibited promise in computer vision tasks, but few works have considered applying it to IML. To bridge this gap, this paper introduces a neural architecture search (NAS) approach integrated with transformer [neural architecture search and transformer (NAST)] for IML. The proposed NAST framework incorporates three key components: (1) a fusion layer composed of a standard convolutional layer, a Bayar convolutional layer, and a steganalysis-rich model layer to capture multi-perspective features; (2) two NAS modules—one repeated six times and the other four times—automatically constructed from a pool of neural operators; and (3) a Vision Transformer block positioned between the two NAS modules for enhanced feature processing. Experimental results demonstrate that the proposed NAST achieves excellent localization performance on multiple datasets. The proposed NAST highlights the potential of combining NAS with transformer mechanisms, paving the way for further advancements in media forensics.