Increasing thermal performance and preventing heat loss are very important in energy conversion systems, especially for new and complex products that exacerbate this need. Therefore, to solve this challenge, a trapezoidal cavity with a wavy top wall containing water/ethylene glycol GO–Al2O3 nanofluid is simulated using Galerkin finite element method. The effects of physical parameters affecting thermal performance and fluid flow, including porosity (ℇ), thermal radiation (Rd), magnetic field angle (α), Rayleigh number (Ra) and Hartmann number (Ha), are investigated in the determined ratios. The results of applied boundary conditions showed that the optimal values for Ra, Ha, ℇ, Rd and α are 1214.46, 2.86, 0.63, 0.24 and 59.35, respectively. Considering that changes in radiation have little effect on streamlines and isothermal lines. Optimization by RSM and Taguchi integration resulted in optimal Nu detection. It provided a correlation for the average Nu based on the investigated determinants due to the conflicting influence of the study factors, which finally calculated the highest average Nusselt number of 3.07. Therefore, the ideal design, which is the primary goal of this research, increases the thermal performance.