Mathematical Perspectives of a Coupled System of Nonlinear Hybrid Stochastic Fractional Differential Equations

This research develops a novel coupled system of nonlinear hybrid stochastic fractional differential equations that integrates neutral effects, stochastic perturbations, and hybrid switching mechanisms. The system is formulated using the Atangana–Baleanu–Caputo fractional operator with a non-singular Mittag–Leffler kernel, which enables accurate representation of memory effects without singularities. Unlike existing approaches, which are limited to either neutral or hybrid stochastic structures, the proposed framework unifies both features within a fractional setting, capturing the joint influence of randomness, history, and abrupt transitions in real-world processes. We establish the existence and uniqueness of mild solutions via the Picard approximation method under generalized Carathéodory-type conditions, allowing for non-Lipschitz nonlinearities. In addition, mean-square Mittag–Leffler stability is analyzed to characterize the boundedness and decay properties of solutions under stochastic fluctuations. Several illustrative examples are provided to validate the theoretical findings and demonstrate their applicability.