Mutual Learning across heterogeneous architectures for image classification

For image classification, feature-level mutual learning has remained largely limited to models from the same architecture family. Because the classification objective maps the entire input image to a single class probability distribution, each architecture independently optimizes its intermediate representations according to its distinct inductive biases. This causes heterogeneous feature spaces to become misaligned and incompatible for direct feature matching. Consequently, methods that successfully exchange intermediate features across heterogeneous architectures have remained limited to dense prediction tasks, such as semantic segmentation, where the objective compels intermediate representations to preserve the original image geometry. We introduce a model-agnostic approach for image classification that aligns both intermediate representations and final probability distributions across heterogeneous architectures, surpassing classical logit-based mutual learning by nearly 8 percentage points while remaining competitive with strong offline distillation.Our method is straightforward: a 50%-masked image passes through a context encoder, while the full image passes through the target. A predictor then maps the context features to the target’s representations at masked positions. This prediction operates bidirectionally across both models. The contrastive loss enables distinct architectures to transfer relational knowledge through relative similarities in their latent representations rather than direct point-to-point matching of intermediate feature maps. We instantiate the method on a heterogeneous pairing of ResNet-18 and ViT-Small, presenting substantial inductive-bias differences, trained from scratch on Tiny-ImageNet.