In recent years, the interest of female dominance in long-distance swimming has grown where several newspaper articles have been published speculating about female performance and dominance—especially in open-water ultra-distance swimming. The aim of this narrative review is to review the scientific literature regarding the difference between the sexes for all swimming strokes (i.e., butterfly, backstroke, breaststroke, freestyle and individual medley), different distances (i.e., from sprint to ultra-distances), extreme conditions (i.e., cold water), different ages and swimming integrated in multi-sports disciplines, such as triathlon, in various age groups and over calendar years. The influence of various physiological, psychological, anthropometrical and biomechanical aspects to potentially explain the female dominance was also discussed. The data bases Scopus and PUBMED were searched by April 2020 for the terms ’sex–difference–swimming’. Long-distance open-water swimmers and pool swimmers of different ages and performance levels were mainly investigated. In open-water long-distance swimming events of the ’Triple Crown of Open Water Swimming’ with the ’Catalina Channel Swim’, the ’English Channel Swim’ and the ’Manhattan Island Marathon Swim’, women were about 0.06 km/h faster than men. In master swimmers (i.e., age groups 25–29 to 90–94 years) competing in the FINA (Fédération Internationale de Natation) World Championships in pool swimming in freestyle, backstroke, butterfly, breaststroke, individual medley and in 3000-m open-water swimming, women master swimmers appeared able to achieve similar performances as men in the oldest age groups (i.e., older than 75–80 years). In boys and girls aged 5–18 years—and listed in the all-time top 100 U.S. freestyle swimming performances from 50 m to 1500 m—the five fastest girls were faster than the five fastest boys until the age of ~10 years. After the age of 10 years, and until the age of 17 years, however, boys were increasingly faster than girls. Therefore, women tended to decrease the existing sex differences in specific age groups (i.e., younger than 10 years and older than 75–80 years) and swimming strokes in pool-swimming or even to overperform men in long-distance open-water swimming (distance of ~30 km), especially under extreme weather conditions (water colder than ~20 °C). Two main variables may explain why women can swim faster than men in open-water swimming events: (i) the long distance of around 30 km, (ii) and water colder than ~20 °C. Future studies may investigate more detailed (e.g., anthropometry) the very young (<10 years) and very old (>75–80 years) age groups in swimming