Visualization based malware classification under extreme class imbalance: a transfer learning approach with systematic ablation

Abstract Malware detection and classification continues to be a major concern of cybersecurity since threat attackers are refining to come up with more advanced and elusive malicious code. Visualization-based solutions that transform raw binary executables into grayscale images and the use of convolutional neural networks to classify them have shown excellent potential. However, little is known about the influence of each training element in a real-world classification state of class imbalance. This work presents a transfer-learned EfficientNet-B0 pipeline with a grayscale-adapted convolutional stem, trained on the Microsoft BIG 2015 benchmark of 10,868 binary visualizations spanning nine malware families. A controlled ablation across six variants was conducted to isolate the contribution of each training component. The components under test are inverse-frequency class weighting, a WeightedRandomSampler, ImageNet pretraining, and the choice between weighted CrossEntropy and Focal Loss. The full pipeline reaches $$98.82 \pm 0.07$$ test accuracy and $$97.06 \pm 0.51$$ macro-F1 across five seeds, with perfect recall on the 7-sample Simda minority class. The sampler and class-weighted loss carry minority-class recovery as complementary mechanisms, pretraining mainly speeds convergence, and Swapping in Focal Loss at = 0.5 collapses macro-F1 by more than ten points through the complete loss of the minority Simda class, an effect isolated to the loss function itself rather than attributed to a specific mechanism. A Friedman omnibus test across five seeds rejects equality of the variants on accuracy, macro-F1 and calibration error, and bootstrap confidence intervals together with large Cliff’s delta effect sizes place the full pipeline above every ablated variant on every seed. The pairwise Wilcoxon and McNemar tests do not reach conventional significance after Holm correction, since a five-seed budget bounds the smallest attainable adjusted p-value at 0.156; the component rankings are therefore reported as consistent and large in effect size rather than as formally significant pairwise results. Grad-CAM and LIME show the model attends to genuine byte-level structures, not augmentation artifacts. The suggested framework provides a practically useful, interpretable and computationally limited malware family classification pipeline using visualization as the classification technique, and provides practical instructions to practitioners on using such systems within resource-constrained security environments.