Masked Autoencoder with Noise-Adaptive Gated Attention for Multimodal Cultural Relic Image Inpainting

Traditional mural image inpainting methods mostly rely on visual information from known image areas, suffering from weak semantic understanding, missing damaged content, and low consistency between restoration outputs and textual semantic descriptions. To tackle these challenges, this paper designs a Masked Autoencoder equipped with noise-adaptive gated attention (MAE GA) for multimodal cultural heritage image restoration. Joint weighted optimization is performed over pixel reconstruction, multi-granularity semantic alignment, perceptual quality, and spatial continuity. Evaluated on the Dunhuang mural subset from the CulTi multimodal dataset, our model obtains PSNR = 30.45 dB, SSIM = 0.938, and Recall@1 = 0.658 for the text-image retrieval task. Comprehensive ablation experiments demonstrate that the proposed semantic and spatial constraints consistently improve semantic recall, perceptual quality, and edge continuity, while introducing an interpretable trade-off in pixel-level PSNR/SSIM: the configuration without alignment achieves higher PSNR/SSIM, but exhibits substantially lower Recall@1, poorer FID, and weaker edge continuity. Five-fold cross-validation, repeated random-seed experiments, and robustness experiments under multiple degradation patterns further demonstrate the model's stable generalization capability. Subjective evaluations by experts confirm the high texture fidelity and semantic plausibility of the restored outputs. This study further examines performance degradation when textual descriptions are inaccurate, incomplete, ambiguous, or unavailable, and discusses the corresponding limitations. The proposed algorithm provides a feasible technical reference for multimodal image restoration of cultural heritage artifacts.

Authors

Publication Details

Journal
International Journal of Pattern Recognition and Artificial Intelligence
Published
2026-11-19
DOI
https://doi.org/10.1142/s021800142640080x
Primary Topic
Generative Adversarial Networks and Image Synthesis
Type
article
Field-Weighted Citation Impact
0.00
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

Masked Autoencoder with Noise-Adaptive Gated Attention for Multimodal Cultural Relic Image Inpainting

Long Zhang, Chenye Zhang
International Journal of Pattern Recognition and Artificial Intelligence
Generative Adversarial Networks and Image Synthesis
article

Masked Autoencoder with Noise-Adaptive Gated Attention for Multimodal Cultural Relic Image Inpainting

Long Zhang, Chenye Zhang
article en

Abstract

Traditional mural image inpainting methods mostly rely on visual information from known image areas, suffering from weak semantic understanding, missing damaged content, and low consistency between restoration outputs and textual semantic descriptions. To tackle these challenges, this paper designs a Masked Autoencoder equipped with noise-adaptive gated attention (MAE GA) for multimodal cultural heritage image restoration. Joint weighted optimization is performed over pixel reconstruction, multi-granularity semantic alignment, perceptual quality, and spatial continuity. Evaluated on the Dunhuang mural subset from the CulTi multimodal dataset, our model obtains PSNR = 30.45 dB, SSIM = 0.938, and Recall@1 = 0.658 for the text-image retrieval task. Comprehensive ablation experiments demonstrate that the proposed semantic and spatial constraints consistently improve semantic recall, perceptual quality, and edge continuity, while introducing an interpretable trade-off in pixel-level PSNR/SSIM: the configuration without alignment achieves higher PSNR/SSIM, but exhibits substantially lower Recall@1, poorer FID, and weaker edge continuity. Five-fold cross-validation, repeated random-seed experiments, and robustness experiments under multiple degradation patterns further demonstrate the model's stable generalization capability. Subjective evaluations by experts confirm the high texture fidelity and semantic plausibility of the restored outputs. This study further examines performance degradation when textual descriptions are inaccurate, incomplete, ambiguous, or unavailable, and discusses the corresponding limitations. The proposed algorithm provides a feasible technical reference for multimodal image restoration of cultural heritage artifacts.

International Journal of Pattern Recognition and Artificial Intelligence
Generative Adversarial Networks and Image Synthesis
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon