Sex differences in recombination are widespread in mammals, but the causes of this pattern are poorly understood. Previously, males from two interfertile subspecies of house mice, Mus musculus musculus and M. m. castaneus, were shown to exhibit a ∼30% difference in their global crossover frequencies. Much of this crossover rate divergence is explained by six autosomal loci and a large-effect locus on the X chromosome. Intriguingly, the allelic effects at this X-linked locus are transgressive, with the allele conferring increased crossover rate being transmitted by the low crossover rate M. m. castaneus parent. Despite the pronounced divergence between males, females from these subspecies exhibit similar crossover rates, raising the question of how recombination is genetically controlled in this sex. Here, I analyze publicly available genotype data from early generations of the Collaborative Cross, an eight-way panel of recombinant inbred strains, to estimate crossover frequencies in female mice with sex-chromosome genotypes of diverse subspecific origins. Consistent with the transgressive influence of the X chromosome in males, I show that females inheriting an M. m. castaneus X possess higher average crossover rates than females lacking the M. m. castaneus X chromosome. The differential inheritance of the X chromosome in males and females provides a simple genetic explanation for sex-limited evolution of this trait. Further, the presence of X-linked and autosomal crossover rate modifiers with antagonistic effects hints at an underlying genetic conflict fueled by selection for distinct crossover rate optima in males and females.