Sex is determined by chromosomes in mammals but it can be influenced by the environment in many worms, crustaceans, and vertebrates. Despite this, there is little understanding of the relationship between ecology and the evolution of sexual systems. The nematode Auanema freiburgensis has a unique sex determination system in which individuals carrying one X chromosome develop into males while XX individuals develop into females in stress‐free environments and self‐fertile hermaphrodites in stressful environments. Theory predicts that trioecious populations with coexisting males, females, and hermaphrodites should be unstable intermediates in evolutionary transitions between mating systems. In this article, we study a mathematical model of reproductive evolution based on the unique life history and sex determination of A. freiburgensis. We develop the model in two scenarios, one where the relative production of hermaphrodites and females is entirely dependent on the environment and one based on empirical measurements of a population that displays incomplete, “leaky” environmental dependence. In the first scenario environmental conditions can push the population along an evolutionary continuum and result in the stable maintenance of multiple reproductive systems. The second “leaky” scenario results in the maintenance of three sexes for all environmental conditions. Theoretical investigations of reproductive system transitions have focused on the evolutionary costs and benefits of sex. Here, we show that the flexible sex determination system of A. freiburgensis may contribute to population‐level resilience in the microscopic nematode's patchy, ephemeral natural habitat. Our results demonstrate that life history, ecology, and environment may play defining roles in the evolution of sexual systems.

Sex is determined by chromosomes in mammals but it can be influenced by the environment in many worms, crustaceans and vertebrates. Despite this, there is little understanding of the relationship between ecology and the evolution of sexual systems. The nematode Auanema freiburgensis has a unique sex determination system in which individuals carrying one X chromosome develop into males while XX individuals develop into females in stress-free environments and self-fertile hermaphrodites in stressful environments. Theory predicts that trioecious populations 42 Kanzaki et al., 2017). Auanema is a recently described nematode genus with two trioecious species, 43A. rhodensis and A. freiburgensis (Kanzaki et al., 2017) and is distantly related to the trioecious 44 Heterorhabditis sp. (Strauch et al., 1994). 45 3 In this article, we develop a new mathematical model to study how the flexible sex determi-46