This paper aims to obtain a better heat transfer performance by using vortex generators with different structures in the square channel. To compare the flow pattern of three different vortex generators, the stream‐wise velocity profile, the visualized Q‐criterion, and the stream‐wise vorticity flux distribution were used to obtain the flow structures in detail. The heat transfer performance was quantified by the Nusselt number (Nu) and compared for the different vortex generators. It was found that the concave structure vortex generator has a better heat transfer performance with an increase of about 11.07% for average Nu. This is because the concave vortex generator induces a more complex hydraulic dynamic flow structure due to its two counter rotating vortex pairs in the square channel. Furthermore, the distance from the lower counter rotating vortex pairs to the wall is significantly decreased by the restraint from the upper counter rotating vortex pairs. This can help disturb the thermal boundary layer and improve the heat transfer performance.