The evolutionary transition from hermaphroditism (combined sexes) to dioecy (separate sexes) is associated with whole genome duplication (polyploidy) in several flowering plant genera. Moreover, there is evidence for transitions in the opposite direction, i.e. a loss of dioecy with an increase in ploidy. Here, we review evidence for these associations, synthesize previous ideas on the mechanism underlying the patterns and explore alternative pathways. Specifically, we examine potential ecological and genetic mechanisms, differentiated by whether ploidy or gender (functional sex expression of the plant) changes are the primary cause and whether the effect is direct or indirect. An analysis of 22 genera variable for both ploidy and gender indicates that gender monomorphism (hermaphroditism, monoecy) is more common among diploid than polyploid species, whereas gender dimorphism (dioecy, gynodioecy, subdioecy) is more frequent among polyploid species. The transition from diploid hermaphroditic to polyploid gender-dimorphic taxa may arise directly through changes in gender as a result of genome duplication through genomic rearrangements or homeologous recombination, or changes in gender may result in increased unreduced gamete production leading to polyploid formation. Alternatively, the transition may occur through the indirect effects of genome duplication on mating system and inbreeding depression, which favor selection for unisexuality, or habitat shifts associated with unisexuality may simultaneously cause increased unreduced gamete production. Novel mechanisms for transitions in the opposite direction (from dioecy to hermaphroditism with increase in ploidy) include disruption of genetic sex determination and the benefits of reproductive assurance. We highlight key questions requiring further attention and promising approaches for answering them and better clarifying the genesis of sexual system polyploidy associations. See also the sister article focusing on animals by Wertheim et al. in this themed issue.