Mutual Learning across heterogeneous architectures for image classification

For image classification, feature-level mutual learning has remained largely limited to models from the same architecture family. Because the classification objective maps the entire input image to a single class probability distribution, each architecture independently optimizes its intermediate representations according to its distinct inductive biases. This causes heterogeneous feature spaces to become misaligned and incompatible for direct feature matching. Consequently, methods that successfully exchange intermediate features across heterogeneous architectures have remained limited to dense prediction tasks, such as semantic segmentation, where the objective compels intermediate representations to preserve the original image geometry. We introduce a model-agnostic approach for image classification that aligns both intermediate representations and final probability distributions across heterogeneous architectures, surpassing classical logit-based mutual learning by nearly 8 percentage points while remaining competitive with strong offline distillation. At each step, both models execute two forward passes: a full-image target pass and a masked context pass. The full-image pass projects intermediate representations into a shared latent space to yield ground-truth target tokens. In the masked context pass, a shared binary mask hides the same half of the patches for both models, and each model projects the features of this partial view into the same space as the target tokens, yielding context tokens. Each architecture then predicts, from its visible context tokens, the peer’s target tokens at those hidden positions. This prediction, which is evaluated using a contrastive loss, operates bidirectionally across both models. We instantiate the method on a heterogeneous pairing of ResNet-18 and ViT-Small, presenting substantial inductive-bias differences, trained from scratch on Tiny-ImageNet. For questions or collaboration inquiries, please contact me at: [email protected]