In recent years, due to the low thermal coefficients of common fluids and the increase in size and cost of heat exchangers, technologies for improving heat transfer and reducing dimensions have been developed and widely used in industries such as refrigeration, cooling of processing cells, chemical industries, and more. Previously, increasing heat exchange capacity in heat exchangers was achieved by altering parameters such as boundary conditions, flow geometry, heat exchanger geometry, or changing the type of fluid. Additionally, apart from the use of nanofluids, various other operational methods can be employed to improve the thermal performance of heat exchangers. Accordingly, considering the combination of the aforementioned innovative techniques, this study presents the modeling of flow and heat transfer inside helically coiled tube heat exchangers under the flow of nanofluids containing nickel, gold, silver, and gold/silver hybrid nanoparticles using numerical and artificial intelligence methods. In this study, the effect of variations in the inner diameter of the coiled tube and the volume fraction of nanoparticles was examined. The results showed that increasing the inner diameter and volume fraction of nanoparticles leads to an increase in heat transfer coefficient and Nusselt number, while the friction factor decreases with an increase in Reynolds number and increases with an increase in diameter and volume fraction. Finally, the accuracy and validity of the model were evaluated using statistical parameters and experimental results, which showed a 99.9% level of agreement between the predicted and experimental outcomes.