XGBoost Web Bot Detection Using Session-Level Features from HTTP Access Logs

Automated web traffic includes malicious programs that carry out denial-of-service attacks, data scraping, and credential stuffing, so separating bot sessions from human sessions is useful for protecting web servers. This study examined how well session-level features computed only from web server access logs, without biometric data or client network addresses, could separate bot sessions from human sessions. Raw logs from a public benchmark dataset were parsed into 833 sessions (409 human and 424 bot), and 24 features describing request timing, endpoint access, response behavior, and User-Agent characteristics were extracted. Random forest and extreme gradient boosting (XGBoost) classifiers were trained with fixed hyperparameters and without resampling, using a stratified 80:20 train-test split and five-fold stratified cross-validation. On the test set of 167 sessions, XGBoost reached an accuracy of 94.61%, a harmonic mean of precision and recall of 94.74%, and an area under the receiver operating characteristic curve of 97.58%, while random forest reached 94.61%, 94.67%, and 97.36%. The two models gave different predictions for only two test sessions, and cross-validation accuracy was 94.60% for XGBoost and 95.08% for random forest. Features derived from the User-Agent string had the highest gain-based importance in XGBoost. The XGBoost model was also implemented as a prototype with a backend service and a frontend dashboard, and five black-box test scenarios passed. These results come from a single controlled benchmark, were obtained by classifying complete sessions offline, and require validation on other data.