Enhancing Botnet Detection with Machine Learning and Explainable AI: A Step Towards Trustworthy AI Security

The rapid proliferation of botnets, armies of compromised machines controlled by malicious actors remotely, has played apivotal role in the increase in cyber-attacks, such as Distributed Denial-of-Service (DDoS) attacks, credential theft, dataexfiltration, command-and-control (C2) activity, and automated exploitation of vulnerabilities. Legacy botnet detectionmethods, founded on signature matching and deep packet inspection (DPI), are rapidly becoming a relic of the past becauseof the prevalence of encryption schemes like TLS 1.3, DNS-over-HTTPS (DoH), and encrypted VPN tunneling. Theseencryption mechanisms conceal packet payloads, making traditional network monitoring technology unsuitable for botnetdetection. Faced with the challenge, ML-based botnet detection mechanisms have risen to the top. Existing ML-basedapproaches, however, are marred by two inherent weaknesses: (1) Lack of granularity in detection because most models arebased on binary classification, with no distinction of botnet attack variants, and (2) Uninterpretability, where highperforming AI models behave like black-box mechanisms, which limits trust in security automation and leads to high falsepositives, thereby making threat analysis difficult for security practitioners.To overcome these challenges, this study proposes an AI-based, multi-class classification botnet detection system forencrypted network traffic that includes Explainable AI (XAI) techniques for improving model explainability and decisiontransparency. Two datasets, CICIDS-2017 and CTU-NCC, are used in this study, where a systematic data preprocessingstep was employed to maximise data quality, feature representation, and model performance. Preprocessing includedduplicate record removal, missing and infinite value imputation, categorical feature transformation, and removal of highlycorrelated and zero-variance features to minimise model bias. Dimensionality reduction was performed using PrincipalComponent Analysis (PCA), lowering features of CICIDS-2017 from 70 to 34 and those of CTU-NCC from 17 to 4 formaximizing computational efficiency. Additionally, to deal with skewed class distributions, Synthetic Minority OverSampling Technique (SMOTE) was employed to synthesise minority class samples to offer balanced representation of botnetattack types For CICIDS-2017, we used three machine learning algorithms: Random Forest (RF) with cross-validation (0.98 accuracy,100K samples per class), eXtreme Gradient Boosting (XGB) with Bayesian optimisation (0.997 accuracy, 180K samples perclass), and our recently introduced Hybrid K-Nearest Neighbours(KNN) + Random Forest (RF) model, resulting in stateof-the-art accuracy of 0.99 (180K samples per class). The CTU-NCC dataset was divided across three network sensors andprocessed separately. Random Forest (RF), Decision Tree (DT), and KNN models were trained independently for eachsensor, and to enhance performance, ensemble learning methods such as stacking and voting were applied to combine theresults from each of the sensors. The resulting accuracies were as follows: (Random Forest Stacking: 99.38%, RandomForest Voting: 99.35% ), (Decision Tree Stacking: 99.68%, Decision Tree Voting: 91.65%), and (KNN Stacking: 97.53%,KNN Voting: 97.11%). Explainable AI (XAI) techniques like SHAP (SHapley Additive exPlanations) and LIME (LocalInterpretable Model agnostic Explanation) were integrated to provide enhanced interpretability in eXtreme GradientBoosting and our Hybrid KNN+Random Forest model, which provided explanations for model decisions and enhancedanalyst confidence in the system prediction.Our key contribution is the Hybrid KNN+Random Forest system with 0.99 accuracy and provision of explainability. Weillustrate an accurate, scalable, and deployable AI-based solution for botnet attacks. Our experimentation shows that themulti-class classification method greatly assists in botnet attack discrimination, and Explainable AI (XAI) helps enhanceclarity and is thus a strong, practical solution in the real case of botnet detection in an encrypted network scenario.