To investigate the heat transfer coefficient (HTC) of a newly developed rare-earth wrought magnesium alloy under different cooling rates, the experiment of solution treatment followed by water quenching or air cooling process was carried out for calculation by lumped capacitance method (LCM) and optimized by inverse heat transfer method (IHTM), and cooling temperature curves were simulated afterward. In water quenching, the larger the temperature difference between the sample and water, the larger the maximum HTC, and the earlier it reached the maximum value, and in air cooling the HTC became larger with the airflow speeds increased. In LCM, the peak values of the HTC were 2840 W/(m2·°C) in water quenching and 54 W/(m2·°C) in air cooling. The corresponding HTC was 2388 W/(m2·°C) in IHTM. The maximum absolute average relative error (AARE) of temperature simulation in water quenching decreased from 8.46% in LCM to 2.45% in IHTM. The residual stress(RS) of a large conical component was simulated using both non-optimized and optimized HTC, the RS in the IHTM was ~30 MPa smaller than that in the ILCM, because the corresponding HTC was smaller, and the comparison of the simulation results with the measurements revealed that the RS using HTC in the IHTM is more accurate.