Physical dormancy of seeds is a form of dormancy due to the presence of an impermeable seed coat layer, and it represents a feature for plants to adapt to environmental changes over an extended period of phylogenetic evolution. However, in agricultural practice, physical dormancy is problematic. because it prevents timely and uniform seed germination. Therefore, physical dormancy is an important agronomical trait to target in breeding and domestication, especially for many leguminous crops. Compared to the well-characterized physiological dormancy, research progress on physical dormancy at the molecular level has been limited until recent years, due to the lack of suitable research materials. This review focuses on the structure of seed coat, factors affecting physical dormancy, genes controlling physical dormancy, and plants suitable for studying physical dormancy at the molecular level. Our goal is to provide a plethora of information for further molecular research on physical dormancy.