The microstructure and impact toughness of weld metals (WMs) and heat-affected zones (HAZs) of a low-alloy CrMoV steel gas turbine rotor which had served for 14 years were investigated. The ex-service joints in the turbine part (serving at 500–540 °C) and the compressor part (serving below 300 °C) of the rotor were selected for comparative research. The microstructure of the WMs and HAZs between the turbine part and the compressor part was similar, indicating that there was no significant deterioration in microstructure of the turbine part during service. However, compared with the compressor part WM, the impact energy of the turbine part WM decreased significantly, and FATT50 increased greatly. The degraded toughness of turbine part WM was related to more serious intergranular cracking caused by higher segregation level of phosphorus (P) at prior austenite grain (PAG) boundaries. Welding and post-weld heat treatment led to obvious segregation of P at PAG boundaries in WMs, and the segregation of P in turbine part WM was further aggravated during serving at 500–540 °C. Additionally, the inhomogeneous microstructure of the WMs also aggravated the segregation of P. The toughness of the HAZs in both turbine part and compressor part was high, which was because of fine grains. Furthermore, due to there being more grain boundaries and low P content, the segregation of P in HAZs was slight and its adverse effect on the toughness could be ignored.