Polyploidy, the result of whole-genome duplication (WGD), is a major driver of eukaryote evolution. All angiosperms have undergone ancient WGDs, and today stable polyploids account for a substantial portion of both wild and domesticated plant species. Despite its centrality in evolution, WGD is a hugely disruptive mutation, and we still lack a clear understanding of its fitness consequences. Here, we study whether WGD results in greater diversity of genomic structural variants (SVs) and how this influences evolutionary dynamics in nature. Using a set of long-read sequenced samples from the plant genusCochlearia(Brassicaceae), which contains diploids and a recent ploidy series up to octoploid, we show that masking of recessive mutations due to WGD has led to a substantial and progressive accumulation of genic SVs across four ploidy levels. Such SVs likely constitute a genetic load and thus reduce the adaptive potential of polyploid populations. However, this SV accumulation also provides a rich pool of standing genetic variation upon which selection may act in novel environments. By constructing a graph-based pangenome forCochlearia, we identify SVs in hundreds of samples and study the genomic basis of environmental adaptation. We find putatively beneficial SVs involved in pathogen resistance, root development, and salt tolerance, many of which are unique to polyploids. Finally, we explore the adaptive landscapes of SVs and SNPs, identify geographical regions where SVs make novel contributions to adaptive variation, and predict that the role of SVs in environmental adaptation increases due to rapid climate change.