Observation of a zero-energy excitation mode in the open Dicke model

Approaching phase boundaries in many-body systems can give rise to intriguing signatures in their excitation spectra. Here, we explore the excitation spectrum of a Bose-Einstein condensate strongly coupled to an optical cavity and pumped by an optical standing wave, which simulates the famous Dicke-Hepp-Lieb phase transition of the open Dicke model with dissipation arising due to photon leakage from the cavity. For weak dissipation, the excitation spectrum displays two strongly polaritonic modes. Close to the phase boundary, we observe an intriguing regime where the lower energetic of these modes, instead of showing the expected roton-type mode softening, is found to approach and persist at zero energy, well before the critical pump strength for the Dicke-Hepp-Lieb transition boundary is reached. Hence, a peculiar situation arises, where an excitation is possible at zero energy cost, but nevertheless no instability of the system is created.