Generalized Gradient Estimation for Variational Autoencoders and Reinforcement Learning

This work presents a generalized gradient estimator that optimizes expectations involving known or black-box functions for discrete and continuous random variables. We synthesize and extend standard methods for constructing gradient estimators, offering a framework that incurs minimal computational overhead. Our proposed approach demonstrates effectiveness in variational autoencoders and introduces a straightforward extension to reinforcement learning, accommodating discrete and continuous action settings. Experiment-tal results reveal improved training performance and sample efficiency, highlighting the utility of our estimator in various domains. Future applications include training models with complex attention mechanisms, continuous latent-variable models with non-differentiable likelihoods, and integrating our method with existing variance-reduction techniques and optimization methods in reinforcement learning.