Additive genetic variation for fitness at vulnerable life stages governs the adaptive potential of populations facing stressful conditions under climate change, and can depend on current conditions as well as those experienced by past stages or generations. For sexual populations, fertilization is the key stage that links one generation to the next, yet the effects of fertilization environment on the adaptive potential at the vulnerable stages that then unfold during development are rarely considered, despite climatic stress posing risks for gamete function and fertility in many taxa and external fertilizers especially. Here, we develop a simple fitness landscape model exploring the effects of environmental stress at fertilization and development on the adaptive potential in early life. We then test our model with a quantitative genetic breeding design exposing family groups of a marine external fertilizer, the tubeworm Galeolaria caespitosa, to a factorial manipulation of current and projected temperatures at fertilization and development. We find that adaptive potential in early life is substantially reduced, to the point of being no longer detectable, by genotype‐specific carryover effects of fertilization under projected warming. We interpret these results in light of our fitness landscape model, and argue that the thermal environment at fertilization deserves more attention than it currently receives when forecasting the adaptive potential of populations confronting climate change.