ORL-LO: Offline Reinforcement Learning for Pretraining and Finetuning in Logic Optimization

As a key step in digital integrated circuit (IC) design, logic synthesis involves various logic optimization algorithms, where the quality of results (QoR) depends heavily on the optimization sequence used. Exploring the optimization space is challenging as the number of potential optimal permutations grows exponentially. Traditional methods rely on manual adjustments by experts, but are difficult to deal with complex and different circuits, leading to significant optimality gaps. Many automatic methods have been introduced, but they still face problems of low generalization and low efficiency. In this work, we propose ORL-LO, a two-stage learning framework that leverages an offline reinforcement learning (RL) method (i.e., advantage-weighted actor-critic, AWAC) to pretrain on expert trajectories and finetune on target test circuits, enabling efficient exploration of optimal synthesis flows. Firstly, ORL-LO uses AWAC to pretrain a fast and high-performance intelligent policy to fully leverage the offline knowledge of a large corpus of high-quality expert trajectories. Then, the pretrained policy is finetuned for target circuits using AWAC and policy distillation to obtain better results. Compared with the state-of-the-art (SOTA) method, our framework can effectively improve the quality of logic optimization and significantly speed up the exploration time.