Gamete compatibility, and fertilization success, is mediated by gamete‐recognition genes (GRGs) that are expected to show genetic evidence of a response to sexual selection associated with mating system traits. Changes in the strength of sexual selection can arise from the resolution of sperm competition among males, sexual conflicts of interest between males and females, or other mechanisms of sexual selection. To assess these expectations, we compared patterns of episodic diversifying selection among genes expressed in the gonads of Cryptasterina pentagona and C. hystera, which recently speciated and have evolved different mating systems (gonochoric or hermaphroditic), modes of fertilization (outcrossing or selfing), and dispersal (planktonic larvae or internal brooding). Cryptasterina spp. inhabit the upper intertidal of the coast of Queensland and coral islands of the Great Barrier Reef. We found some evidence for positive selection on a GRG in the outcrossing C. pentagona, and we found evidence of loss of gene function in a GRG of the self‐fertilizing C. hystera. The modification or loss of gene functionality may be evidence of relaxed selection on some aspects of gamete interaction in C. hystera. In addition to these genes involved in gamete interactions, we also found genes under selection linked to abiotic stress, chromosomal regulation, polyspermy, and egg‐laying. We interpret those results as possible evidence that Cryptasterina spp. with different mating systems may have been adapting in divergent ways to oxidative stress or other factors associated with reproduction in the physiologically challenging environment of the high intertidal.Research HighlightsRecent speciation between two sea stars was unlikely the result of selection on gamete‐recognition genes annotated in this study. Instead, our results point to selection on genes linked to the intertidal environment and reproduction.