Drilling fluid is indispensable to assure the safety and success of a drilling operation. Besides the normal drilling fluid such as water-based mud or oil-based mud, a new kind of drilling fluid has emerged recently, which consisted of the use of nanotechnology. The aim of this paper is to study the cooling effect of nano-drilling fluid used in the petroleum industry. A dynamic model that included a reservoir formation, a well, and a drill string in the drilling process with drilling fluid circulation was built for this objective. Navier-Stoke equation was used for the fluid flow inside the well and the drill string, while Darcy’s equation was used for the flow inside the formation. The rise of temperature due to friction was also accounted for in this model. Two types of drilling fluid were used in the simulation: the normal drilling fluid and the one using nanotechnology. The change of temperature in the wellbore and in the formation over time with these two types of drilling fluid was observed at various positions: at the bottom hole where the drilling bit is constantly in contact with the formation, and at other places further away from the bottom hole. The simulated results showed that, although the temperature fluctuated in the two cases but on average, the nano drilling fluid gave a better cooling effect in comparison with the normal one. This article is the first study about the application of nanoparticles in drilling fluid in Vietnam using an integrated modeling method. The approach proposed in this article can be applied efficiently in practical applications of nano drilling fluid for petroleum drilling in Vietnam. However, it is noted that this research treated typically the technical side of the application of nanotechnology in drilling fluid, while it will be necessary to asset the financial aspect in order to make this technology a real-life application.