Fatality rates of infectious diseases are often higher in men than women. Although this difference is often attributed to a stronger immune response in women, we show that differences in the transmission routes that the sexes provide can result in evolution favouring pathogens with sex-specific virulence. Because women can transmit pathogens during pregnancy, birth or breast-feeding, pathogens adapt, evolving lower virulence in women. This can resolve the long-standing puzzle on progression from Human T-cell Lymphotropic Virus Type 1 (HTLV-1) infection to lethal Adult T-cell Leukaemia (ATL); a progression that is more likely in Japanese men than women, while it is equally likely in Caribbean women and men. We argue that breastfeeding, being more prolonged in Japan than in the Caribbean, may have driven the difference in virulence between the two populations. Our finding signifies the importance of investigating the differences in genetic expression profile of pathogens in males and females.