Understanding the nature of the magnetic coupling in transition metal doped <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:msub><mml:mrow><mml:mi>Bi</mml:mi></mml:mrow><mml:mn>2</mml:mn></mml:msub><mml:msub><mml:mrow><mml:mi>Se</mml:mi

In this paper, we employ electronic structure theory to investigate the nature of the exchange coupling in transition metal doped <a:math xmlns:a="http://www.w3.org/1998/Math/MathML"><a:mrow><a:msub><a:mi>Bi</a:mi><a:mn>2</a:mn></a:msub><a:msub><a:mi>Se</a:mi><a:mn>3</a:mn></a:msub></a:mrow></a:math>. We focus on V, Cr, Mn, and Fe, which have been under scrutiny for the realization of the quantum anomalous Hall effect. For simplicity, we model the doping process as happening inside a single layer of Bi within a single quintuple layer inside a three-formula-unit conventional hexagonal cell and consider situations of full coverage or half coverage. Due to the covalent nature of the chemical bond between transition metal atoms and selenium, this simple model is capable of describing the fundamental features of the intraplane exchange coupling, while offering a clearer analysis of the response to structural, magnetic, and electronic perturbations. In agreement with recent literature, our results confirm that the van Vleck mechanism has a very marginal contribution to the exchange coupling. Depending on the filling of the <b:math xmlns:b="http://www.w3.org/1998/Math/MathML"><b:mrow><b:mn>3</b:mn><b:mi>d</b:mi></b:mrow></b:math> shell and on the details of the electronic structure, several other mechanisms compete and cooperate to induce the magnetic order. For V, double exchange and superexchange cooperate to have the strongest ferromagnetic coupling among the investigated elements, followed by Cr where only superexchange is active. For Mn, superexchange and double exchange compete to create an extremely weak ferromagnetic order. For Fe, a strong antiferromagnetic coupling caused by the double exchange is observed to dominate over the ferromagnetic Ruderman-Kittel-Kasuya-Yosida interaction, but the high sensitivity of this competition to the doping concentration suggests that our conclusion may not hold in the dilute limit. Overall, Fe doping seems to offer the most intriguing competition of exchange mechanisms that can be tuned by adjusting the doping concentration and the details of the host, as, e.g., by replacing Se with Te or Bi with Sb. Published by the American Physical Society 2024