A comparative study was conducted to reveal the effect of microstructure on hydrogen permeation in the EA4T and 30CrNiMoV12 railway axle steels. Unlike the EA4T with its sorbite structure, 30CrNiMoV12 steel shows a typical tempered martensitic structure, in which a large number of fine, short, rod-like, and spherical carbides are uniformly dispersed at boundaries and inside laths. More importantly, this structure possesses plentifully strong hydrogen traps, such as nanosized Cr7C3, Mo2C, VC, and V4C3, thus resulting in a high density of trapping sites (N = 1.17 × 1022 cm−3). The hydrogen permeation experiments further demonstrated that, compared to EA4T, the 30CrNiMoV12 steel not only delivered minimally effective hydrogen diffusivity but also had a high hydrogen concentration. The activation energy for hydrogen diffusion of the 30CrNiMoV12 steel was greatly increased from 23.27 ± 1.94 of EA4T to 47.82 ± 2.14 kJ mol−1.