Artificial neural network and response surface method-based optimization of unsteady natural convective heat transfer of Fe3O4-Engine Oil nanofluid in a trapezoidal enclosure

• A mathematical correlation for the average Nussle number has been built using response surface methodology. • The ANN model has a superior R 2 and reduced MSE and RMSE values, signifying enhanced overall performance as compared to the RSM model. • The artificial intelligence techniques significantly reduce the computational time, from approximately 7 hours for CFD simulations to just 5 minutes for ANN predictions. • The highest value of the average Nu was found at ϕ = 0.04, Ha = 10 and Ra = 10 6 for both ANN and RSM methods. This study focuses on optimizing the heat transport characteristics of the natural convection of Fe 3 O 4 -engine oil nanofluid in a trapezoidal domain under the influence of a sloping periodic magnetic field. The optimization process employs response surface methodology (RSM), and artificial neural networks (ANN) based on a one-component thermally equilibrium mathematical model. The governing dimensionless equations and boundary conditions are numerically solved by the Galerkin-weighted residual FEM. The numerical results are used to train and evaluate the performance of both RSM and ANN models. A central composite design within the RSM framework is applied for statistical experimental design, exploring combinations of input parameter values. Sensitivity analysis is used to evaluate how input parameters affect the response that is produced. Low Hartmann number ( Ha ), high Rayleigh number ( Ra ), and large nanoparticle volume fractions ( ϕ ) are the conditions that produced the best thermal performance. Several statistical indices, including the coefficient of determination (R²), mean squared error (MSE), margin of deviation (MOD), and root mean squared error (RMSE), are used to quantitatively calculate the quality of the RSM and ANN models. The results show that both RSM and ANN are convenient approaches for estimating the Fe₃O₄-EO nanofluid's HT rate, but that the ANN performs better than RSM in terms of correlation and prediction accuracy. The findings demonstrate how ANN models may be used as reliable instruments to examine the thermal behaviour of industrial applications that use nanofluids as operational fluids. The optimum value of the heat transfer rate is found at ϕ = 0.04, Ha = 10, and Ra = 10 6 for both ANN and RSM methods.