Leveraging recurrence in neural network wavefunctions for large-scale simulations of Heisenberg antiferromagnets on the triangular lattice

Variational Monte Carlo simulations have been crucial for understanding quantum many-body systems, especially when the Hamiltonian is frustrated and the ground-state wavefunction has a nontrivial sign structure. In this paper, we use recurrent neural network (RNN) wavefunction ans"atze to study the triangular-lattice antiferromagnetic Heisenberg model (TLAHM) for lattice sizes up to $30\ifmmode\times\else\texttimes\fi{}30$. In a recent study [M. S. Moss et al., Phys. Rev. B 112, 134450 (2025)], the authors demonstrated how RNN wavefunctions can be iteratively retrained in order to obtain variational results for multiple lattice sizes with a reasonable amount of compute. That study, which looked at the sign-free, square-lattice antiferromagnetic Heisenberg model, showed favorable scaling properties, allowing accurate finite-size extrapolations to the thermodynamic limit. In contrast, our present results illustrate in detail the relative difficulty in simulating the sign-problematic TLAHM. We find that the accuracy of our simulations can be significantly improved by transforming the Hamiltonian with a judicious choice of basis rotation. We also show that a similar benefit can be achieved by using variational neural annealing, an alternative optimization technique that minimizes a pseudo-free energy. Ultimately, we are able to obtain estimates of the ground-state properties of the TLAHM in the thermodynamic limit that are in close agreement with values in the literature, showing that RNN wavefunctions provide a powerful toolbox for performing finite-size scaling studies for frustrated quantum many-body systems.