As the water pipe cooling system is widely applied to controlling temperature in mass concrete structures, the precise simulation of the temperature field in mass concrete with cooling pipes embedded is meaningful. This paper presents an isogeometric analysis (IGA) with NURBS for heat transfer in mass concrete with consideration of the cooling pipe. The proposed method not only achieves the same level of accuracy with fewer nodes but also eliminates the time-consuming process of mesh in the traditional FEM. The coarsest parameter space which depicts small pipe and large concrete precisely is constructed to create an efficient model for numerical computation. In addition, the unique k-refinement in IGA is supposed to be the most appropriate encryption mechanism, and the knot insertion vector for effective refinement is calculated by considering the characteristics of temperature gradient distribution around the cooling pipes. In addition, a different calculation parameter has been discussed to show the stability and flexibility of the IGA. The obtained numerical results demonstrate the accuracy and efficiency of the proposed scheme in the simulation of transient temperature fields in concrete structures with cooling systems.