The strategic location of North Africa has led to cultural and demographic shifts, shaping its genetic structure. Historical migrations brought different genetic components that are evident in present-day North African genomes, along with autochthonous components. The Imazighen (plural of Amazigh) are believed to be the descendants of autochthonous North Africans and speak various Amazigh languages, which belong to the Afro-Asiatic language family. However, the arrival of different human groups, especially during the Arab conquest, caused cultural and linguistic changes in local populations, increasing their heterogeneity. We aim to characterize the genetic structure of the region, using the largest Amazigh dataset to date and other reference samples. Our findings indicate microgeographical genetic heterogeneity among Amazigh populations, modeled by various admixture waves and different effective population sizes. A first admixture wave is detected group-wide around the twelfth century, whereas a second wave appears in some Amazigh groups around the nineteenth century. These events involved populations with higher genetic ancestry from south of the Sahara compared to the current North Africans. A plausible explanation would be the historical trans-Saharan slave trade, which lasted from the Roman times to the nineteenth century. Furthermore, our investigation shows that assortative mating in North Africa has been rare.