Southern Arabia and the Horn of Africa are important geographic centers for the study of human population history because a great deal of migration has characterized these regions since the first emergence of humans out of Africa. Analysis of Jewish groups provides a unique opportunity to investigate more recent population histories in this area. Mitochondrial DNA is used to investigate the maternal evolutionary history and can be combined with historical and linguistic data to test various population histories. In this study, we assay mitochondrial control region DNA sequence and diagnostic coding variants in Yemenite (n = 45) and Ethiopian (n = 41) Jewish populations, as well as in neighboring non-Jewish Yemeni (n = 50) and Ethiopian (previously published Semitic speakers) populations. We investigate their population histories through a comparison of haplogroup distributions and phylogenetic networks. A high frequency of sub-Saharan African L haplogroups was found in both Jewish populations, indicating a significant African maternal contribution unlike other Jewish Diaspora populations. However, no identical haplotypes were shared between the Yemenite and Ethiopian Jewish populations, suggesting very little gene flow between the populations and potentially distinct maternal population histories. These new data are also used to investigate alternate population histories in the context of historical and linguistic data. Specifically, Yemenite Jewish mitochondrial diversity reflects potential descent from ancient Israeli exiles and shared African and Middle Eastern ancestry with little evidence for large-scale conversion of local Yemeni. In contrast, the Ethiopian Jewish population appears to be a subset of the larger Ethiopian population suggesting descent primarily through conversion of local women.