Dual Masking for Mutual Masked Autoencoderefficient Feature Extraction Without Large-to-Small Compression

With the rise of LLMs and MLLMs, pre-training large models and fine-tuning them for specific tasks have become common. However, this approach faces challenges such as high computational costs, the need for task-specific fine-tuning, and difficulties in deploying large models, especially in Edge Computing and Embodied Intelligence. To address these issues, we propose Dual Masking for Mutual-Masked Autoencoder (DMMAM), a novel framework that enhances feature extraction without traditional large-to-small model compression. DMMAM employs two identical models that collaborate through mutual learning to improve feature representation. By designing distinct masking maps, the encoders extract complementary features through distance minimization between encoded representations, enabling mutual reference and knowledge sharing. Additionally, to prevent decoders from being misled by the original input, they exclusively decode jointly masked tokens while integrating features from the parallel encoder for deeper interactive learning. Experiments on ImageNet demonstrate that our approach achieves feature representation quality comparable to pre-trained large-scale models while maintaining a compact structure, thus making it particularly suitable for edge computing and efficient model distillation in embodied intelligence.