Nanofluids in minichannels with various configurations are applied as cooling and heating fluids. Therefore, it is essential to have an optimal design of minichannels. For this purpose, a square channel with a cylinder in the center connected to wavy fins at various concentrations of an Al2O3 nanofluid is simulated using the finite volume method (FVM). Moreover, central composite design (CCD) is used as a method of design of experiment (DOE) to study the effects of three input variables, namely the cylinder diameter, channel width, and fin radius on the convective heat transfer and pumping power. The impacts of the linear term, together with those of the square and interactive on the response variables are determined using Pareto and main effects plots by an ANOVA. The non-dominated sorting genetic algorithm-II (NSGA-II), along with the response surface methodology (RSM) is applied to achieve the optimal configuration and nanofluid concentration. The results indicate that the effect of the channel width and cylinder diameter enhances about 21% and 18% by increasing the concentration from 0% to 5%. On the other hand, the pumping power response is not sensitive to the nanofluid concentration. Besides, the channel width has the highest and lowest effect on the heat transfer coefficient (HTC) and pumping power, respectively. The optimization for a concentration of 3% indicates that in Re = 500 when the geometry is optimized, the HTC enhances by almost 9%, while the pumping power increases by about 18%. In contrast, by increasing the concentration from 1% to 3%, merely an 8% enhancement in HTC is obtained, while the pumping power intensifies around 60%.