Phishing URL Detection Using XGBoost and Custom Feature Engineering

Abstract: Phishing is a prevalent cyberattack technique that deceives users into revealing sensitive personal and financial information through fake websites. With the exponential growth of online services, phishing attacks have become more sophisticated, necessitating intelligent and automated detection mechanisms. This study introduces a smart phishing URL detection approach that utilizes carefully engineered features—such as lexical patterns, structural elements, and domain-related information—to differentiate between malicious and legitimate web addresses. A custom feature extraction module was developed to parse URLs and retrieve 13+ critical features, including URL length, directory structure, file name characteristics, presence of IP addresses, SSL certificate availability, information about the Autonomous System Number (ASN) and domain registration details, including creation and expiration dates. The extracted features were used to train an Extreme Gradient Boosting (XGBoost) classifier, selected for its superior performance in imbalanced and noisy datasets. The model was developed and fine-tuned using PyCaret, an automated machine learning library that optimizes classification performance using cross-validation and hyperparameter tuning. The trained model achieved strong performance across multiple evaluation metrics, highlighting its reliability and effectiveness in accurately identifying phishing URLs. To enhance usability, a web-based application was developed using FastAPI and HTML/CSS, allowing users to submit a URL and receive instant predictions regarding its legitimacy. The system provides an interpretable and scalable framework for real-time phishing detection, suitable for integration into email filters, browsers, and cybersecurity tools. The results affirm that combining feature engineering with a tuned XGBoost classifier offers an effective and deployable solution to mitigate phishing threats in real-world environments