Adaptation to contrasting environments occurs when advantageous alleles accumulate in each population, but it remains largely unknown whether these same advantageous alleles create genetic incompatibilities that can cause intrinsic reproductive isolation leading to speciation. Identifying alleles that underlie both adaptation and reproductive isolation is further complicated by factors such as dominance and genetic interactions among loci, which can affect both processes differently and obscure potential links between adaptation and speciation. Here, we use a combination of field and glasshouse experiments to explore the connection between adaptation and speciation while accounting for dominance and genetic interactions. We created a hybrid population with equal contributions from four contrasting ecotypes of Senecio lautus (Asteraceae), which produced hybrid genomes both before (F1 hybrid generation) and after (F4 hybrid generation) recombination among the parental ecotypes. In the glasshouse, plants in the second generation (F2 hybrid generation) showed reduced fitness as a loss of fertility. However, fertility was recovered in subsequent generations, suggesting that genetic variation underlying the fitness reduction was lost in subsequent generations. To quantify the effects of losing genetic variation at the F2 generation on the fitness of later generation hybrids, we used a reciprocal transplant to test for fitness differences between parental ecotypes, and F1 and F4 hybrids in all four parental habitats. Compared to the parental ecotypes and F1 hybrids, variance in F4 hybrid fitness was lower, and lowest in habitats that showed stronger native-ecotype advantage, suggesting that stronger natural selection for the native ecotype reduced fitness variation in the F4 hybrids. Fitness trade-offs that were present in the parental ecotypes and F1 hybrids were absent in the F4 hybrid. Together, these results suggest that the genetic variation lost after the F2 generation was likely associated with both adaptation and intrinsic reproductive isolation among ecotypes from contrasting habitats.