Robust Learning-to-Rank Algorithm for Bug Discovery in Hardware Verification

Constrained-random testing is a main method for novel bug discovery in unit-level verification. Its compute intensity inspired machine learning (ML) technique such as reinforcement learning (RL), which showed promising results. However, it is still challenging to productionize RL applications because of design complexity, constant data drift, and expensive training cost. As an alternative, using binary pass-fail data, we previously developed an ML test recommender app that prioritized bug-prone tests. However, the app often chose already-found bugs since the model’s objective function ignored novelty of fail signatures. Here, we propose an update with a learning-to-rank algorithm and bootstrap aggregation (bagging) that prioritizes tests with rare fail signatures over others. Real-world CPU benchmark datasets show the new update finds twice as many rare failures and never-seen novel signatures than the classifier. Further model analysis indicates that the choice of learning-to-rank algorithm over classification and the bagging technique is critical to robust model performance.