Quantum phase transition and exceptional points of a non-Hermitian Hamiltonian for cold atoms in a dissipative optical cavity with nonlinear atom-photon interactions

We investigate in this paper the macroscopic quantum states of multiple atoms in an optical cavity with non-Hermitian coupling motivated by recent experiments. The tunable atom-photon nonlinear interaction is included to demonstrate the variety of quantum phases in the realistic realization of Dicke phase transition with cold atoms. The energies of macroscopic quantum states are obtained in terms of the spin-coherent-state variational method, in which the nonunitary operator of the similarity transformation is used to diagonalize the effective non-Hermitian Hamiltonian, which is pseudo-Hermitian below the exceptional point (EP). The superradiant phase (SP) and related quantum phase transition (QPT) disappear completely for the non-Hermitian Dicke model (DM) because of imaginary coupling, which leads to instability of states with nonzero photons. The situation is dramatically changed by including the nonlinear atom-photon interaction generated by the pump field. It is found that the second EP emerges at a higher value of coupling. The spectrum is complex between the two EPs and real otherwise. Phase diagrams are obtained for red and blue cavity pump field detunings, showing explicitly the revival of the SP and QPT, respectively. However, the QPT from the SP to the normal phase with the increase of coupling is opposite to the transition in the normal DM. The critical property of the Berry phase indicates the first-order nature of the QPT, which again is different from the DM, where the QPT is known as the second order.