In the present study, the mechanism of improving HAZ toughness of steel plate with Mg deoxidation after the simulated welding with the heat input of 400 kJ/cm was investigated through in situ observation, characterization with SEM-EDS and TEM-EDS, and thermodynamic calculation. It was found that intragranular acicular ferrite (IAF) and polygonal ferrite (PF) contributed to the improvements of HAZ toughness in steels with Mg deoxidation. With the increase of Mg content in steel, the oxide in micron size inclusion was firstly changed to MgO-Ti2O3, then to MgO with the further increase of Mg content in steel. The formation of nanoscale TiN particles was promoted more obviously with the higher Mg content in the steel. The growth rates of austenite grains at the high-temperature stage (1400~1250 °C) during the HAZ thermal cycle of steels with conventional Al deoxidation and Mg deoxidation containing 0.0027 and 0.0099 wt% Mg were 10.55, 0.89, 0.01 μm/s, respectively. It was indicated that nanoscale TiN particles formed in steel with Mg deoxidation were effective to inhibit the growth of austenite grain. The excellent HAZ toughness of steel plates after welding with a heat input of 400 kJ/cm could be obtained by control of the Mg content in steel to selectively promote the formation of IAF or retard the growth of austenite grain.