Fe-30Mn-0.05C steel specimens with cold-rolled, partially recrystallized, fine-grained, and coarse-grained microstructures were fabricated by means of 80% cold rolling followed by annealing at 550–1000 °C. The initial and deformed microstructures were characterized, and the Charpy impact properties were tested at room temperature (RT) and liquid nitrogen temperature (LNT). It was found that the Charpy absorbed energy increased with the annealing temperature, while the specimens showed different trends: parabolic increase at RT and exponential increase at LNT, respectively. Compared with the fully recrystallized specimens, those with a partially recrystallized microstructure exhibited lower impact energy, especially at LNT. This was because cracks tended to nucleate and propagate along the recovery microstructure where stress concentration existed. The grain size played an important role in the twinning behavior and impact properties. High Charpy impact energy (~320 J) was obtained in the coarse-grained specimen having the grain size of 42.1 μm at both RT and LNT, which was attributed to the activation of high-density deformation twinning. However, deformation twinning was inhibited in the specimen with the average grain size of 3.1 μm, resulting in limited work hardening and lower impact energy.