Low-lying electronic structure of the rare earth based topological nodal line semimetal candidate DySbTe

Lanthanide ($Ln$) based $Ln\mathrm{SbTe}$ materials have garnered significant attention due to the rich interplay of long-range magnetic ordering and topological properties, driven by unique crystalline symmetry, $4f$ electron interactions, and pronounced spin-orbit coupling (SOC) effects. DySbTe, as a heavier lanthanide based member of the $Ln\mathrm{SbTe}$ family, stands out with its SOC and larger on-site interactions on its $4f$ electrons, which arise due to the heavier Dy element. Here, we present a comprehensive study on the low-temperature bulk physical properties and the electronic structure of DySbTe using magnetic susceptibility, heat capacity, and electrical resistivity measurements, along with high-resolution angle-resolved photoemission spectroscopy (ARPES), scanning tunneling microscopy and spectroscopy (STM/S), and density functional theory calculations. Our thermodynamic measurements revealed an antiferromagnetic ordering below ${T}_{\mathrm{N}}=7.45$ K and a subsequent magnetic phase transition at ${T}_{\mathrm{N}1}=7.15$ K. Our transport studies indicate a semimetallic behavior with unusual feature in the ordered state. Our ARPES measurements revealed a diamond-shaped Fermi pocket centered at the $\overline{\mathrm{\ensuremath{\Gamma}}}$ point, with band features that evolve distinctly across various binding energies. STM/S results indicate a minimum in the density of states at around 100 meV below the Fermi level, and ARPES measurements reveal a significant gap present around the $\overline{\mathrm{X}}$ point, differentiating DySbTe from other $Ln\mathrm{SbTe}$ compounds. These findings enhance our understanding of the SOC effects on the electronic structure and topological properties in the $Ln\mathrm{SbTe}$ family, highlighting DySbTe as a promising candidate for exploring the interplay between topology and magnetism.