The microbiome may serve as a reservoir of adaptive potential if hosts can leverage microbial adaptations in stressful environments. However, the facilitation of rapid host adaptation may be limited by interactions between host genotype, microbiome, and environment (GH x GM x E). Here, to understand how host x microbiome x environment interactions shape adaptation, we leverage >150 generations of experimental evolution in Drosophila melanogaster in a stressful environment, high sugar (HS) diet. The microbiome of HS adapted flies shifted to be dominated by one bacteria, Acetobacter pasteurianus. We next performed a fully reciprocal transplant experiment using the dominant control and HS bacteria to measure life-history and metabolic traits in flies. Mismatches between fly evolution and microbiome exerted fitness costs by slowing development and reducing fecundity, especially in the stressful HS diet. The GH x GM x E interactions observed suggest that both ecological context and host evolution shape the adaptive potential of the microbiome. We conclude by proposing future directions that highlight the benefit of using experimental evolution to study the contribution of the microbiome to host evolution.