Addressing modulational instability in anti-resonant hollow-core fibers for pulse compression

When pulses propagate in gas-filled anti-resonant hollow-core fibers (AR-HCFs) modulational instability (MI) can lead to pulse break-up and loss of coherence. In pulse broadening and compression schemes, MI is a parasitic effect that induces significant shot-to-shot fluctuations of the peak power of compressed pulses and increases rapidly over a narrow range of input pulse energies. In this work, we use experimental studies and supporting numerical simulations to compare two AR-HCFs that are chosen to enhance or suppress MI. We demonstrate that judicious selection of the wall thickness of the anti-resonant elements (AREs) can drastically reduce the MI gain, thereby increasing the limit of pulse energy scaling of stable ultrafast pulse compression.