Application of different types of nanoparticles including Al2O3, SiO2, and CuO is studied to find the effects of nanoparticles on the thermoeconomic optimization of a fin and tube heat exchanger. The effectiveness and the total annual cost are considered as objective functions, and optimization is performed using multi‐objective particle swarm optimization algorithm by selection of nine design parameters including nanoparticle volumetric concentration and parameters related to the heat exchanger geometry. The effect of nanoparticles on total annual cost (TAC) vs effectiveness indicates that TAC increases by increase of effectiveness. The results also reveal that in all the studied cases, thermoeconomic improvement is found in the case of nanoparticle additive compared with the base fluid for the high heat exchanger effectiveness. Moreover, application of Al2O3 and CuO is more beneficial than SiO2 in the aspect of thermoeconomic point of view. Furthermore, Al2O3 causes a higher effectiveness in the case of considering nanoparticle price, while in the case of neglecting nanoparticle price, CuO is the best for the higher effectiveness. © 2017 Curtin University and John Wiley & Sons, Ltd.