Interplay of canted antiferromagnetism and nematic order in Mott insulating Sr2Ir1-xRhxO4

Sr2IrO4 is one of the prime candidates for realizing exotic quantum spin orders owing to the subtle combination of spin-orbit coupling and electron correlation. Sensitive local magnetization measurement can serve as a powerful tool to study these kinds of systems with multiple competing spin orders since the comprehensive study of the spatially-varying magnetic responses provides crucial information on their energetics. Here, using sensitive magneto-optical Kerr effect measurements and spin Hamiltonian model calculations, we show that Sr2IrO4 has non-trivial domain structures which cannot be understood by conventional antiferromagnetism. This unconventional magnetic response exhibits broken symmetry along the Ir-O-Ir bond direction and is enhanced upon spin-flip transition or Rh-doping. Our analysis, based on possible stacking patterns of spins, shows that introduction of an additional rotational-symmetry breaking is essential to describe the magnetic behavior of Sr2Ir1-xRhxO4, providing strong evidence for a nematic hidden order phase in this highly correlated spin-orbit Mott insulator. Antiferromagnetic order in iridates arising from unusual combinations of strong electron-correlation and spin-orbit coupling has drawn broad attention in the condensed matter community. In this work, the authors uncover the interplay between antiferromagnetic and nematic orders, highlighting its fundamental importance and potential functional applications.