Loss-Guided Diffusion For General Controllable Generation With Sequential Monte Carlo

There has been a recent surge in employing conditional diffusion models across various applications, primarily owing to their superior generative capabilities. However, training-based conditional generation methods lack flexibility and require costly retraining for new conditions to yield high performance, while training-free methods suffer from inconsistency and instability due to heuristic approximations of the conditional score function. To address these challenges, we propose a novel training-free controllable generation approach for diffusion models based on sequential Monte Carlo (SMC), which implements asymptotically exact loss-guided diffusion sampling. We leverage pre-trained encoders to model the likelihood function based on similarity in feature space and energy-based functions and enable general controllable generation across a broad range of conditions. Experimental results demonstrate that the proposed method achieves state-of-the-art performance in both conditional generation quality and condition alignment and outperforms existing training-free conditional generation techniques.