Backpropagation Through Compiled Code

Algorithmic differentiation (AD) tools provide accurate derivatives of computer-implemented functions, for use in gradient-based optimization tasks like the training of neural networks or the improvement of computational engineering designs. Typically, AD tools require the source code to be available in a restricted set of programming languages, and often necessitate manual modifications because the language standard is only partially supported. In contrast, Derivgrind , a novel AD tool that inserts forward-mode differentiation logic on the machine code level via dynamic binary instrumentation, supports an unprecedented range of programming languages with minimal manual integration efforts. In this work, we extend Derivgrind with tape-recording logic, enabling reverse-mode AD for compiled code.