With the environmental pollution and the shortage of oil resources becoming more and more serious, the development and application of new energy vehicles have attracted more and more attention. Engine is an important part of new energy vehicles, and its performance has a great impact on the vehicle. Compared with traditional industrial motors, new energy vehicle engines have higher requirements on power density, and the improvement of power density poses new challenges to the design of motor cooling system. The purpose of this paper is to study the engine cooling device of new energy vehicles and improve the overall performance of the vehicle. The main research content of this paper is to lay a foundation for the theoretical basis of the engine cooling device, elaborate the working principle of the motor cooling system and the loss of the motor in operation. Then, the heat dissipation system of permanent magnet synchronous motor based on heat pipe is studied experimentally. Aiming at the problem of only considering the temperature rise and ignoring the pressure loss in the flow channel design, a flow channel design method considering the motor temperature rise and the flow channel pressure loss is proposed, and the motor flow channel is optimized. The test results show that the maximum temperature rise at the end is close to 16.56 °C, which is in good agreement with the simulation results. It shows that the heat pipe based heat dissipation system can effectively reduce the temperature rise of motor winding, which provides a new idea for the heat dissipation design of permanent magnet synchronous motor