During the early stages of speciation, genetic differences tend to accumulate at certain regions of the genome leading to the formation of genomic islands of divergence (GIDs). This pattern may be due to selection and/or difference in the rate of recombination. Here, we investigate the possible causes of GIDs in Drosophila yakuba mayottensis, a subspecies of Drosophila yakuba specializing on toxic noni (Morinda citrifolia) fruits on the island of Mayotte. Field collection reconfirmed the association of this subspecies with noni fruits. Population genomics did not find evidence for genetic structure on the island, but fine-scale analysis comparing these sequences to two mainland D. y. yakuba populations identified multiple structured genomic regions. We assembled a new genome for D. y. mayottensis using a combination of long and short reads. We identified six chromosomal inversions in high recombining regions, of which two were unique to D. y. mayottensis. Genome-wide differentiation analyses revealed that 92% of outlier genes overlapped with the inversions. We also compared population genomics sequences of Drosophila sechellia, the only other noni-specialist, to its generalist homosequential relatives of the simulans clade. Although we identified several common outlier genes between D. sechellia and D. y. mayottensis, we did not detect traces of GIDs in this clade. Thus, different genomic architectures can underlie complex convergence in response to common selection in independent lineages, leading to distinct genomic divergence patterns.