Bone, as a mineralized composite of inorganic (mostly carbonated hydroxyapatite) and organic (mainly type I collagen) phases, possesses a unique combination of remarkable strength and toughness. Its excellent mechanical properties are related to its hierarchical structures and precise organization of the inorganic and organic phases at the nanoscale: Nanometer-sized hydroxyapatite crystals periodically deposit within the gap zones of collagen fibrils during bone biomineralization process. This hierarchical arrangement produces nanomechanical heterogeneities, which enable a mechanism for high energy dissipation and resistance to fracture. The excellent mechanical properties integrated with the hierarchical nanostructure of bone have inspired chemists and material scientists to develop biomimetic strategies for artificial bone grafts in tissue engineering (TE). This critical review provides a broad overview of the current mechanisms involved in bone biomineralization, and the relationship between bone hierarchical structures and the deformation mechanism. Our goal in this review is to inspire the application of these principles toward bone TE.