The mammalian sperm midpiece has a unique double-helical structure called the mitochondrial sheath that wraps tightly around the axoneme. Despite the remarkable organization of the mitochondrial sheath, the molecular mechanisms involved in mitochondrial sheath formation are unclear. In the process of screening testis-enriched genes for functions in mice, we identified armadillo repeat-containing 12 (ARMC12) as an essential protein for mitochondrial sheath formation. Here, we engineered Armc12-null mice, FLAG-tagged Armc12 knock-in mice, and TBC1 domain family member 21 (Tbc1d21)-null mice to define the functions of ARMC12 in mitochondrial sheath formation in vivo. We discovered that absence of ARMC12 causes abnormal mitochondrial coiling along the flagellum, resulting in reduced sperm motility and male sterility. During spermiogenesis, sperm mitochondria in Armc12-null mice cannot elongate properly at the mitochondrial interlocking step which disrupts abnormal mitochondrial coiling. ARMC12 is a mitochondrial peripheral membrane protein and functions as an adherence factor between mitochondria in cultured cells. ARMC12 in testicular germ cells interacts with mitochondrial proteins MIC60, VDAC2, and VDAC3 as well as TBC1D21 and GK2, which are required for mitochondrial sheath formation. We also observed that TBC1D21 is essential for the interaction between ARMC12 and VDAC proteins in vivo. These results indicate that ARMC12 uses integral mitochondrial membrane proteins VDAC2 and VDAC3 as scaffolds to link mitochondria and works cooperatively with TBC1D21. Thus, our studies have revealed that ARMC12 regulates spatiotemporal mitochondrial dynamics to form the mitochondrial sheath through cooperative interactions with several proteins on the sperm mitochondrial surface.