A Reinforcement Learning-Based Approach for Determining Infeasible Paths of Programs

Program path analysis is an essential component of software defect detection and quality assurance. Accurately identifying infeasible paths can prevent false positives caused by invalid paths, enabling developers to pinpoint actual defects more efficiently and enhancing overall software quality and reliability. This paper proposes an integrated approach for determining infeasible paths based on program path features and constraint-based reinforcement learning. First, a loop-structure path search and reduction algorithm is proposed to systematically simplify path explosion induced by loops. Then, a global subgraph-based path reduction algorithm is introduced to effectively remove redundant and irrelevant paths. Subsequently, we propose a path set generation algorithm guided by control and implication relationships to construct an optimized path set. Path constraints and symbolic path constraints are used to enhance semantic representation. Finally, a reinforcement learning-based model utilizing reachability rewards and exploration rewards to dynamically determine path reachability. Experimental results show that our proposed approach significantly reduces path explosion, accurately identifies infeasible paths and outperforms existing methods in terms of accuracy and computational efficiency.