Code Size Optimization Method based on Reinforcement Learning

Reducing code size is critical for software systems with limited storage. The open-source compiler LLVM provides compilation option sequences that generate binaries of varying sizes when applied to intermediate representations (IRs). Recently, reinforcement learning (RL) has been introduced to automatically search for optimal option sequences. However, existing methods often overlook the compilation behavior of individual options, resulting in inaccurate state representations. They also suffer from large action spaces, which ignore the usefulness, noise robustness, and empirical effectiveness of the generated sequences, limiting the optimization performance. To address these issues, we propose the Code Size Optimization method based on RL (CSORL), which includes two key innovations: State Representation Capturing Compilation Behavior (CSORL-S) and Action Space Modeling with High Utility, Low Noise, and Strong Evidence (CSORL-A). CSORL-S uses effective options as supervision signals to guide IR feature extraction. It also employs CNNs to focus on local features that are critical to option effectiveness, enhancing state accuracy. CSORL-A extracts code-size-reducing samples from a large IR and -Oz sequence combination space. It filters out ineffective options, constructs a weighted option dependency graph using IR improvement frequency as evidence, and applies clustering to build a compact, high-quality action space. Experiments on 6 benchmarks show that CSORL achieves an average code size reduction rate of 15.66%, outperforming Poset-RL by 41.08%. It also achieves the best results across all 6 benchmarks. Ablation studies further confirm the effectiveness of both CSORL-S and CSORL-A.