Numerical and AI-based investigation of natural convection in a nanofluid-saturated elliptical cavity with heterogeneous porous medium

Purpose This study aims to investigate the impact of Soret and Dufour effects on double-diffusive natural convection within an elliptical cavity filled with nanofluid and saturated by a heterogeneous porous medium. The objective is to uncover the coupled transport mechanisms under varying thermophysical and geometric conditions and to evaluate the predictive power of AI-based modeling in this complex domain. Design/methodology/approach A two-dimensional, unsteady, laminar nanofluid flow is simulated using the Incompressible Smoothed Particle Hydrodynamics (ISPH) method, capturing both thermal and solutal buoyancy forces. A heterogeneous porous structure with dual elliptical boundaries is considered to reflect realistic internal gradients. Key dimensionless parameters – Soret number, Dufour number, Rayleigh number, Darcy number and nanoparticle volume fraction – are systematically varied. To complement the numerical solver, an Extreme Gradient Boosting (XGBoost) regression model is trained on the simulation data set to predict the average Nusselt and Sherwood numbers, providing a surrogate model for real-time transport estimation. Findings The results reveal that the Soret and Dufour effects play critical roles in enhancing convective coupling within heterogeneous porous enclosures. Increasing the Soret number from 0.1 to 2 enhances heat and mass transfer rates by approximately 40%, while a higher Darcy number increases fluid motion by 55%. Similarly, the buoyancy ratio (n = 5) leads to a 60% increase in flow velocity. An increase in nanoparticle volume fraction improves thermal conductivity by 30% but also suppresses flow speed by 20% due to elevated viscosity. The XGBoost model demonstrates exceptional predictive performance with R2 > 0.99 and minimal mean squared error, validating its effectiveness in capturing the complex parametric trends generated by ISPH simulations. These quantitative outcomes establish key transport-performance relationships and highlight the novel integration of meshless numerical methods with AI regression. Originality/value To the best of the authors’ knowledge, this work presents the first comprehensive application of ISPH coupled with XGBoost regression to analyze double-diffusive convection influenced by Soret and Dufour effects in a heterogeneous porous elliptical cavity. The quantified parametric insights and the AI-enhanced predictive framework offer a new paradigm for optimizing thermal systems involving porous nanofluid media and complex internal geometries.