Towards Exascale Ab Initio Materials Modelling

Plane-wave density functional theory (DFT) codes, i.e. those using a Fourier basis, consistently account for around a third of the total software usage on ARCHER2. Such codes typically decompose their 3D Fourier domain by distributing pencils of Fourier wave-vectors over MPI processes; 3D Fourier transforms are subsequently usually performed as sets of FFTs of lower dimensionality. The consequent inter-process communications are inherently quadratically-scaling and commonly determine the strong scaling limit of large materials modeling simulations. We present our development of a domain decomposition, implemented in CASTEP (www.castep.org), which, to our knowledge, constitutes a novel approach to communications in the DFT community. We describe our decomposition and load-balancing as a coupled two-dimensional optimisation problem. We explain how a logical 2D process grid effectively achieves linear-scaling communications, and how the consequent constraints on load-balancing are ameliorated by adopting and adapting several task-scheduling algorithms. We illustrate how the development performs in real-world simulations, dramatically reducing communication costs and significantly increasing the strong scaling limit. We illustrate efficient scaling on ARCHER2 to 4x as many processes as previously possible.