TL-RL-FusionNet: Reinforcement Learning-Guided Residual MLP with Fused CNN Embeddings for Efficient and Adaptive Ransomware Detection

Ransomware detection remains challenging because modern variants exhibit diverse, elusive, and partly benign behaviors and can propagate rapidly across interconnected enterprises and sensor-enabled cyber-physical systems, causing cascading operational failures. These characteristics undermine signature-based and static-detection methods. Although machine learning has improved detection, many approaches still rely on fixed objectives that weight samples uniformly, limiting their adaptation to heterogeneity and overlaps between ransomware and benign activities. To address this challenge, we introduce TL-RL-FusionNet, a reinforcement learning (RL)-guided hybrid framework that combines dual transfer learning (TL) backbones, EfficientNetB0 and InceptionV3, with a lightweight residual multi-Layer perceptron (MLP) classifier. The framework converts sandbox reports into RGB grids, extracts features using frozen CNN backbone networks, and fuses embeddings for classification. Training is guided by a tabular Q-learning sample-weighting agent, formulated as a per-sample bandit over discrete weight actions. To prevent cross-fold information leakage, the Q-table is freshly initialized in each cross-validation fold and updated only using the fold-local training partition, whereas the held-out fold is used for the final evaluation. The framework was evaluated using two datasets. On our dataset, TL-RL-FusionNet achieved the best overall performance on Dataset 1, with 99.20% accuracy, 99.40% recall, and 99.84% AUC. On the public EldeRan benchmark, it achieved 90.36% accuracy using the full dynamic feature space and 92.08% using a Mutual Information-selected compact subset. Paired Wilcoxon tests across five folds were used to assess the RL contribution, while additional grid-order sensitivity analysis showed that the image-based representation remained robust under five random 10 × 10 feature-grid permutations. Interpretability analysis using t-distributed stochastic neighbor embedding (t-SNE) and gradient-weighted class activation mapping feature-grid mapping further showed that the model captured discriminative behavioral patterns. Overall, these results demonstrate that RL-guided sample reweighting improves adaptive ransomware detection while maintaining efficiency and interpretability. The dataset and supporting code are publicly available on GitHub.