The non-uniformity of microstructures and mechanical properties across a whole welded joint is a crucial factor leading to its weakening performance and premature failure. Post-weld heat treatment is a primary method for increasing the mechanical properties. However, the evolution mechanism of mechanical properties related to heterogeneous microstructure after heat treatment remains unclear, making it challenging to design the heat treatment process and evaluate its effect comprehensively. In this study, microstructure characterization and a series of mechanical tests of 12Cr1MoV welded joint after the stress relief annealing (SRA) and tempering heat treatment (THT) were conducted. The effect of heat treatment on mechanical properties is analyzed based on the comparison between stress relief annealing and tempering heat treatment in terms of tensile properties, impact toughness, and impact fracture morphology. The results indicate that, after the tempering heat treatment, the evolution of mechanical properties in each subzone of the joint is consistent, i.e., the hardness and tensile strength decreased while the toughness increased. Notably, the most substantial enhancement in toughness is observed in the weld zone, primarily due to a significant reduction in the presence of pre-eutectoid ferrite. Furthermore, it is proved that hardness is an indicator to reflect changes in tensile strength related to the microstructure evolution, which indicates it can be employed to evaluate the effectiveness of post-weld heat treatment in practical engineering.