The impact of genome structure on adaptation is a growing focus in evolutionary biology, revealing an important role for structural variation and recombination landscapes in shaping genetic diversity across genomes and among populations. This is particularly relevant when local adaptation occurs despite gene flow, where clustering of differentiated loci can maintain locally adapted variants by reducing recombination between them. However, the limited genomic resources for non-model species, including reference genomes and recombination maps, has constrained our understanding of these patterns. In this study, we leverage the Atlantic silverside—a non-model fish with extensive local adaptation across a steep latitudinal gradient—as an ideal system to explore how genome structure influences adaptation under varying levels of gene flow, using a newly available reference genome and multiple recombination maps. Analyzing 168 genomes from four populations, we found a continuum of genome-wide differentiation increasing from south to north, reflecting higher connectivity among southern populations and reduced gene flow at northern latitudes. With increasing gene flow, the number and clustering of FSToutlier loci also increased, with differentiated loci tightly clustered in large haploblocks harboring inversions and smaller peaks overlapping putative centromeres. Notably, sequence divergence was only evident in inversions, supporting their role in adaptive divergence with gene flow, whereas centromeres appeared differentiated because of low recombination and reduced diversity, with no indication of elevated sequence divergence. Our results support the hypothesis that clustered genomic architectures evolve with high gene flow and enhance our understanding of how inversions and centromeres are linked to different evolutionary processes.Significance StatementHow populations preserve favorable combinations of genes adapted to their local environment despite reproducing with populations adapted to different conditions is a longstanding question in evolutionary biology. By analyzing the genomes of 168 Atlantic silverside fish from four populations, we found that when populations adapted to different environments frequently interbreed, genetic differences concentrate in specific parts of the genome, particularly in chromosomal inversions—where segments of DNA are flipped. These inversions help preserve locally adapted gene combinations, enabling populations to maintain differences essential for survival in their habitats. This research enhances our understanding of genomic adaptation, a fundamental evolutionary question with increasing relevance as environmental changes pose new challenges globally.