Improving Remote Access Trojans Detection: A Comprehensive Approach Using Machine Learning and Hybrid Feature Engineering

Remote Access Trojans (RATs) pose a serious cybersecurity risk due to their stealthy control over compromised systems. This study presents a detection framework that integrates host, network, and newly engineered behavioral features to enhance the identification of RATs. Two sets of experiments were performed: (i) using the original dataset only, and (ii) using an extended dataset with ten engineered features and importance analysis. The framework was evaluated on a public Kaggle dataset of an RAT and benign traffic. Eight machine learning classifiers were tested, including three baseline methods, four ensemble approaches, and a neural network. Results show that the engineered hybrid feature set substantially improves detection performance. Among the tested algorithms, Random Forest and MLP achieved the strongest performance, with accuracies of 98% and 97%, respectively, while Gradient Boosting and LightGBM also performed competitively. Performance was assessed using multiple metrics, and to gain deeper insight into model learning behavior, learning curves and Precision–Recall curves were analyzed. The results demonstrate how well hybrid feature modeling, neural networks, and ensemble machine learning techniques may improve RAT identification. In future work, exploring the use of explainable ML methods may improve the detection capabilities.