Nonrecombining sex chromosomes are widely found to be more differentiated than autosomes among closely related species, due to smaller effective population size and/or to a disproportionally large-X effect in reproductive isolation. Although fungal mating-type chromosomes can also display large nonrecombining regions, their levels of differentiation compared with autosomes have been little studied. Anther-smut fungi from the Microbotryum genus are castrating pathogens of Caryophyllaceae plants with largely nonrecombining mating-type chromosomes. Using whole genome sequences of 40 fungal strains, we quantified genetic differentiation among strains isolated from the geographically overlapping North American species and subspecies of Silene virginica and S. caroliniana. We inferred that gene flow likely occurred at the early stages of divergence and then completely stopped. We identified large autosomal genomic regions with chromosomal inversions, with higher genetic divergence than the rest of the genomes and highly enriched in selective sweeps, supporting a role of rearrangements in preventing gene flow in genomic regions involved in ecological divergence. Unexpectedly, the nonrecombining mating-type chromosomes showed lower divergence than autosomes due to higher gene flow, which may be promoted by adaptive introgressions of less degenerated mating-type chromosomes. The fact that both mating-type chromosomes are always heterozygous and nonrecombining may explain such patterns that oppose to those found for XY or ZW sex chromosomes. The specific features of mating-type chromosomes may also apply to the UV sex chromosomes determining sexes at the haploid stage in algae and bryophytes and may help test general hypotheses on the evolutionary specificities of sex-related chromosomes.