Optimizers for stabilizing likelihood-free inference

A growing number of applications in particle physics and beyond use neural networks as unbinned likelihood ratio estimators applied to real or simulated data. Precision requirements on the inference tasks demand a high-level of stability from these networks, which are affected by the stochastic nature of training. We show how physics concepts can be used to stabilize network training through a physics-inspired optimizer. In particular, the energy conserving descent (ECD) optimization framework uses classical Hamiltonian dynamics on the space of network parameters to reduce the dependence on the initial conditions while also stabilizing the result near the minimum of the loss function. We develop a version of this optimizer known as <a:math xmlns:a="http://www.w3.org/1998/Math/MathML" display="inline"> <a:mrow> <a:msub> <a:mrow> <a:mi mathvariant="sans-serif">ECD</a:mi> </a:mrow> <a:mrow> <a:mi>q</a:mi> <a:mo>=</a:mo> <a:mn>1</a:mn> </a:mrow> </a:msub> </a:mrow> </a:math> , which has few free hyperparameters with limited ranges guided by physical reasoning. We apply <d:math xmlns:d="http://www.w3.org/1998/Math/MathML" display="inline"> <d:msub> <d:mrow> <d:mi mathvariant="sans-serif">ECD</d:mi> </d:mrow> <d:mrow> <d:mi>q</d:mi> <d:mo>=</d:mo> <d:mn>1</d:mn> </d:mrow> </d:msub> </d:math> to representative likelihood-ratio estimation tasks in particle physics and find on average that it out-performs the widely used Adam optimizer. We expect that ECD will be a useful tool for wide array of data-limited problems, where it is computationally expensive to exhaustively optimize hyperparameters and mitigate fluctuations with ensembling.