The strength of polycrystalline metals increases with a decrease in grain size according to the HallPetch relationship. However, heterogeneous microstructures deviate from this relationship depending on the distribution of grain sizes. This paper introduces a rule of mixtures based approach for determining the characteristic length of the microstructure for heterogeneous weld metal. The proposed grain size parameter, the volume-weighted average grain size, is measured experimentally for nine structural steel weld metals and two base materials. The weld metals are found to have a large variety of grain size distributions that are noticeably broader than those of the base material due to differences in phase contents. The results show that the volume-weighted average grain size is able to capture the influence of grain size distribution on the strength of welded structural steel. Based on the experimental results, a modified Hall-Petch relationship is formulated for the strength prediction of heterogeneous microstructures. The modified relationship is also found to be applicable to data from the literature.