Automated DoS Penetration Testing Using Deep Q Learning Network-Quantile Regression Deep Q Learning Network Algorithms

Penetration test is essential to determine the security level of a network. A penetration test attack path simulates an attack to identify vulnerabilities, reduce likely losses, and continuously enhance security. It helps to facilitate the simulation of different attack scenarios, develops robust security measures, and enables proactive risk assessment. We have combined Mul-VAL with DQN and QR-DQN algorithms to solve the problem of incorrect route prediction and problematic convergence associated with attack path planning training. As a result of this algorithm, an attack tree is generated, paths within the attack graph are searched for, and a deep-first search method is used to create a transfer matrix. In addition, QR-DQN and DQN algorithms determine the optimal attack path for the target system. The results of this study show that although the QR-DQN algorithm requires more resources and takes longer to train than the traditional (DQN) algorithm, it is effective in identifying vulnerabilities and optimizing attack paths.