Machine Learning Based Approach to Threat Classification in Application Security

As the landscape of cyber threats advances at a rapid pace, organizations need to maintain consistent and efficient threat modelling processes within their systems. Traditional threat modeling frameworks such as STRIDE and PASTA rely on manual assessments, which are prone to human error, time-consuming and lack standardization across digital platforms. This study was conducted by building and evaluating four machine learning models that automate the classification of application security threats within an organizational environment, with the goal of enhancing consistency and reducing reliance on manual severity assessments. Using internal data from a security company, two ensemble machine learning algorithms, Random Forest and XGBoost, were implemented and trained on datasets consisting of different feature compositions. The models were later evaluated using four evaluation metrics: accuracy, precision, recall, and F1-Score, through both hold-out validation and 5-fold cross-validation. The model that was trained on five features, including metadata from code review logs, achieved the highest performance with an F1-Score of 95.94%. The inclusion of textual features significantly improved classification performance across all metrics compared to the models that excluded them. Finally, the results of this study demonstrate that machine learning can significantly enhance the accuracy and consistency of threat severity classification, particularly when incorporating contextual features.