In this article, the optimization of a twisted strip heat exchanger was explored aiming at enhancing heat transfer. To this end, governing equations using ANSYS‐FLUENT software and nanofluid flow using the k‐ε RNG turbulence model were simulated in the heat exchanger, respectively. The investigated parameters included temperature, pressure, velocity, turbulence kinetic energy (TKE), nanofluid concentration, nanofluid shape factor, Reynolds number, and the diameter of the spherical obstacle on the twisted tape. In addition, the applied nanoparticles were Ag and TiO2, which were combined as a hybrid with a ratio of 50%–50%. Investigations on the surface of the twisted tape with a spherical obstacle revealed a reduction in the temperature while an increase in pressure changes, velocity, and TKE. Moreover, it was specified that the application of lamina‐shaped (non‐spherical) nanofluid has the maximum convection heat transfer coefficient relative to platelets and spherical shapes. The highest heat transfer rate was achieved at a nanofluid concentration of ϕ=0.06. Based on the findings, the heat transfer rate upgraded by increasing the Reynolds number such that the highest heat transfer rate was gained at Reynolds number 18,000. Finally, the heat transfer coefficient was enhanced by approximately 52% by increasing the diameter of the obstacles from 3 to 12 mm.