Herein, it is aimed to correlate the crystallographic structure, crack propagation, and toughness scatter with the prior austenite grain size (PAGS) in a low‐carbon low‐alloy (LCLA) steel from the perspective of crystallography. Two average austenite grain sizes, 16.1 ± 13.4 μm and 30.1 ± 24.9 μm, are obtained by reheating the experimental steel to 950 °C (S950) and 1100 °C (S1100), respectively. Overall, the weaker variant selection and higher density of high‐angle grain boundaries (HAGBs) are observed in the as‐quenched microstructure of S1100. However, coarser blocks and Bain groups (BPs) also appear in S1100 due to some oversized PAGs, which cause a more uneven microstructure in S1100 than that in S950. The ductile‐to‐brittle transition temperature (DBTT) is estimated to be −60 and −88 °C for S1100 and S950, respectively. In addition, a larger scatter of impact toughness occurs in S1100 in the DBTT range from −35 to −80 °C. The inhomogeneous microstructure inherited from oversized PAGs explains higher DBTT and large scatter in impact toughness of S1100. Therefore, to improve low‐temperature toughness, it is more important to balance the blocks and BPs’ size refinement and uniformity rather than only consider the fraction of HAGBs in quenched LALC steel.