xScore: A Simple Metric for Cross-Domain Robustness in Lightweight Vision Models

Lightweight vision models are widely deployed in mobile and embedded systems, where strict computational and memory budgets demand compact architectures. However, their evaluation remains dominated by ImageNet—a single, large natural-image dataset that requires substantial training resources. This creates a dilemma: lightweight models trained on ImageNet often reach capacity limits due to their constrained size, while scaling them to billions of parameters with specialized training tricks to achieve top-tier ImageNet accuracy does not guarantee proportional performance once the architectures are scaled back down to meet mobile constraints, particularly when re-evaluated on diverse data domains. These challenges raise two key questions: How should cross-dataset robustness be quantified in a simple and lightweight way, and which architectural elements consistently support generalization under tight resource constraints? To answer them, we introduce the Cross-Dataset Score (xScore), a simple metric that captures both average accuracy across domains and the stability of model rankings. Evaluating 11 representative lightweight models (2.5 M parameters) across seven datasets, we find that (1) ImageNet accuracy is a weak proxy for cross-domain performance, (2) xScore provides a simple and interpretable robustness metric, and (3) high-xScore models reveal architectural patterns linked to stronger generalization. Finally, the architectural insights and evaluation framework presented here provide practical guidance for measuring the xScore of future lightweight models.