The drying characteristics of poplar wood (Populus adenopoda Maxim) under three treatments were investigated with the aim of improving the drying quality of the impregnated wood. Untreated materials, impregnated materials with 10% initial moisture content (MC), and impregnated materials with 20% initial MC—denoted as T1, T2, and T3, respectively—were examined during conventional drying. Relative to that of T1, the average drying rate of T2 was reduced by 46%, and that of T3 was reduced by 37%. The anti-swelling efficiency of T2 sharply increased, whereas that of T3 only slightly increased in width and thickness. The drying defects of T2 were markedly less than those of T1 and T3, attributed to the reduced residual stress and fixation of the modifier. Scanning electron microscopy indicated that large amounts of modifiers were deposited inside the cell cavity of T2, whereas small amounts of modifiers were deposited in the cell cavity of T3. The impregnation treatment generally reduced the average drying rate and MC uniformity and delayed the decrease in residual stress; however, it reduced the number of defects in the wood and considerably improved the dimensional stability of the wood. Meanwhile, impregnation treatment with increased initial MC could potentially lead to more drying defects and increase production cost.