High‐field magnets have been used in a wide range of scientific and industrial areas. REBCO (REBa2Cu3Ox, where RE = Rare Earth) coated superconductors (CCs) have been gaining increasing interest due to the potentialities of using them in high‐field magnets. However, the thermal stability of the REBCO CCs coils is limited by longitudinal heat conduction inside the coils and bubbles generated by evaporating helium. So as to enhance the heat transfer in REBCO coil, a cooling system based on GM/JT cycle was designed. In this paper, the design, thermodynamic calculation, and analysis of the cooling system for the REBCO coil are provided systematically. It is found that when the mass flow rate is 0.75 g/s at a pressure of 10 bar, the cooling capacity of the system is up to 10.16 W. Meanwhile, the inlet pressure and counterflow heat exchangers efficiency are key factors for significantly improving the cooling capacity of the system. An open loop experiment also has been carried out and the cooling capacity of 0.79 W at 4.5 K is obtained finally when He mass flow rate is 0.1 g/s. © 2024 Institute of Electrical Engineer of Japan and Wiley Periodicals LLC.