Tiny vs. tinier: Baseline ViT-tiny vs. ensemble-distilled student on imbalanced fracture detection

This study compares the performance between a baseline Vision Transformer (ViT-Tiny) and the ensembled-distilled student model (ViT-Tiny distilled via ResNet-18, MobileNetV3-Large, and EfficientNet-B0 teachers) on the task of bone fracture X-ray image classification with a highly imbalanced dataset (1:5, fracture:non-fracture). We incorporate crucial pipelines, combining Stratified k-fold cross-validation (SKFCV) with inline SMOTE (Synthetic Minority Over-sampling Technique), class weighted losses, numerous data augmentation techniques, mixed-precision training, warmup-cosine learning rate scheduling, and early stopping to tackle the highly nonoptimal dataset. The distilled student model successfully closes the gap with the teacher ensemble while maintaining ViT-Tiny’s compact footprint, achieving an F1-score of 0.7033 and MCC of 0.6433 and surpassing both its CNN teachers and baseline. A transferability experiment on an external dataset evaluated our model’s generalization capabilities of the distilled model.