Characterization of antisite defects and in-gap states in antiferromagnetic MnSb₂Te₄

Abstract Intrinsic magnetic topological insulators have been reported to exhibit novel physical phenomena such as the quantum anomalous Hall effect and axion insulator states, demonstrating potential for applications in spintronics and topological quantum computing. Here we perform low-temperature scanning tunneling microscopy (STM) investigations of the antiferromagnetic ground state of MnSb 2 Te 4 , a predicted magnetic topological insulator isostructural with MnBi 2 Te 4 . We visualize the hexagonal Te-terminated surface of MnSb 2 Te 4 and identify two distinct defects originating from different antisite substitutions. Notably, we identify an in-gap state above the Fermi energy where the tunneling spectrum exhibits a negative differential conductance behavior. This electronic state can be modulated by external electric and magnetic fields, suggesting effective pathways for electronic state manipulation. Spin-resolved STM measurements further reveal additional magnetic resonance peaks associated with Mn antisite defects. Our results provide novel insights into the investigation of magnetic topological insulators and demonstrate a promising approach to modulate the localized electronic states.