A Machine Learning-enhanced web-crawler for vulnerability detection : A binary classification approach

Background. The increasing use of web-applications and the Internet underscores the growing importance of application security, as many applications are more vulnerable than anticipated. Penetration testers often employ web-crawling due to its simplicity. However, traditional crawlers struggle with depth, effectiveness, and produce numerous false-positives and negatives. Research on machine learning-induced web crawlers is limited in terms of its ability to enhance the detection of vulnerabilities. Objectives. Machine learning-powered focused crawlers offer significant potential to identify relevant web-pages in response to search queries and efficiently indexing the web. This thesis examines the effectiveness of Random forest, XGBoost, and LSTM-CNN in identifying web-pages that are susceptible to attacks. Methods. In this study, an experimental approach is used to assess the performance differences between a machine learning, ML, integrated web crawler and a conventional one. The performance is assessed by looking at the harvest rate, decreasing the total number of pages saved, and increasing the number of vulnerable pages saved. Results. XGBoost achieved the highest F1-score and recall, while LSTM-CNN scored the lowest among the models evaluated. However, the experiment indicated that many vulnerable pages were overlooked. Conclusions. The top-performing model shows potential, but is not strong enough to be used in production. Currently, ML cannot reliably distinguish vulnerable pages solely through HTML-analysis.