Adaptation to a local environment often occurs in the face of maladaptive gene flow. In this perspective, I discuss several ideas on how a genome may respond to maladaptive gene flow during adaptation. On the one hand, selection can build clusters of locally adaptive alleles at fortuitously co-localized loci within a genome, thereby facilitating local adaptation with gene flow (‘allele-only clustering’). On the other hand, the selective pressure to link adaptive alleles may drive co-localization of the actual loci relevant for local adaptation within a genome through structural genome changes or an evolving intra-genomic crossover rate (‘locus clustering’). While the expected outcome is, in both cases, a higher frequency of locally adaptive alleles in some genome regions than others, the molecular units evolving in response to gene flow differ (i.e., alleles versus loci). I argue that, although making this distinction is important, we commonly lack the critical empirical evidence to do so. This is mainly because many current approaches are biased towards detecting local adaptation in genome regions with low crossover rates. The importance of low-crossover genome regions for adaptation with gene flow, such as in co-localizing relevant loci within a genome, thus remains unclear. Future empirical investigations should address these questions by making use of comparative genomics, where multiple de novo genome assemblies from species evolved under different degrees of genetic exchange are compared. This research promises to advance our understanding of how a genome adapts to maladaptive gene flow, thereby promoting adaptive divergence and reproductive isolation.