Bone tissue engineering integrates biomaterials, cells, and bioactive agents to propose sophisticated treatment options over conventional choices. Scaffolds have central roles in this scenario, and precisely designed and fabricated structures with the highest similarity to bone tissue have shown promising outcomes. On the other hand, using nanotechnology and nanomaterials as the enabling options confers fascinating properties to the scaffolds, such as precisely tailoring the physicochemical features and better interactions with cells and surrounding tissues. Among different nanomaterials, polymeric nanofibers and carbon nanofibers have attracted significant attention due to their similarity to bone extracellular matrix (ECM) and high surface-to-volume ratio. Moreover, bone ECM is a biocomposite of collagen fibers and hydroxyapatite crystals; accordingly, researchers have tried to mimic this biocomposite using the mineralization of various polymeric and carbon nanofibers and have shown that the mineralized nanofibers are promising structures to augment the bone healing process in the tissue engineering scenario. In this paper, we reviewed the bone structure, bone defects/fracture healing process, and various structures/cells/growth factors applicable to bone tissue engineering applications. Then, we highlighted the mineralized polymeric and carbon nanofibers and their fabrication methods.