In this study, vacuum low-pressure carburizing heat treatments were carried out on 18Cr2Ni4WA case-carburized alloy steel. The evolution and phase transformation mechanism of the microstructure of the carburized layer during low-temperature tempering and its effect on the surface hardness were studied. The results showed that the carburized layer of the 18Cr2Ni4WA steel was composed of a large quantity of martensite and retained austenite. The type of martensite matrix changed from acicular martensite to lath martensite from the surface to the core. The hardness of the carburized layer gradually decreased as the carbon content decreased. A thermodynamic model was used to show that the low-carbon retained austenite was easier to transform into martensite at lower temperatures, since the high-carbon retained austenite was more thermally stable than the low-carbon retained austenite. The mechanical stability—not the thermal stability—of the retained austenite in the carburized layer dominated after carburizing and quenching, and cryogenic treatment had a limited effect on promoting the martensite formation. During low-temperature tempering, the solid-solution carbon content of the martensite decreased, the compressive stress on the retained austenite was reduced and the mechanical stability of the retained austenite decreased. Therefore, during cooling after low-temperature tempering, the low-carbon retained austenite transformed into martensite, whereas the high-carbon retained austenite still remained in the microstructure. The changes in the martensite matrix hardness had a far greater effect than the transformation of the retained austenite to martensite on the case hardness of the carburized layer.