A hierarchical deep learning framework with doubly regularized loss for robust malware detection and family categorization

The rapid escalation of cyber threats and the sophistication of malware demand detection frameworks that are accurate, mathematically grounded, and interpretable. Traditional deep learning approaches often rely on standard loss formulations and empirical tuning, leaving challenges such as class imbalance, overfitting, and error propagation insufficiently addressed. This study proposes a hierarchical deep learning framework for Portable Executable (PE) malware detection and family categorization, underpinned by a novel Doubly Regularized Binary Cross-Entropy (DRBCE) loss function. The DRBCE integrates weighted cross-entropy with spectral regularization, and we provide a rigorous mathematical formulation demonstrating its capacity to stabilize training dynamics and mitigate imbalance. The framework employs a two-stage pipeline, first distinguishing benign from malicious files, then categorizing malware into 14 families. We further model error propagation across stages using a probabilistic framework, quantifying how binary classification errors influence downstream family-level predictions. To ensure optimality, hyperparameter tuning is expressed as a constrained optimization problem, systematically solved via KerasTuner. Experimental evaluations on large-scale datasets (BODMAS, SOMLAP, and CLaMP) validate the theoretical analysis, achieving state-of-the-art performance with weighted accuracies up to 99.89% in binary detection and 96.62% in family categorization. Thus, the proposed framework bridges mathematical modeling and applied AI in cybersecurity, offering theoretical insights and practical advances toward robust and interpretable malware detection.