Understanding the genetics of lifetime fitness is crucial to understand a species’ ecological preferences and ultimately predict its ability to cope with novel environmental conditions. Yet, there is a dearth of information regarding the impact of the ecological variance experienced by natural populations on expressed phenotypic and fitness differences. Here, we follow the natural dynamics of experimental A. thaliana populations over 5 successive plantings whose timing was determined by the natural progression of the plant’s life cycle and disentangle the environmental and genetic factors that drive plant ecological performance at a given locality. We show that, at the temperate latitude where the experiment was conducted, a given genotype can experience winter-, spring- or summer-annual life cycles across successive seasons. Lifetime fitness across these seasons varied strongly, with a fall planting yielding 36-fold higher fitness compared to a spring planting. In addition, the actual life-stage at which plant overwinter oscillated across years, depending on the timing of the end of the summer season. We observed a rare but severe fitness differential coinciding with inadequate early flowering in one of the five planting. Substrate variation played a comparatively minor role, but also contributed to modulate the magnitude of fitness differentials between genotypes. Finally, reciprocal introgressions on chromosome 4 demonstrated that the fitness effect of a specific chromosomal region is strongly contingent on micro-geographic and seasonal fluctuations. Our study contributes to emphasize the extent to which the fitness impact of phenotypic traits and the genes that encode them in the genome can fluctuate. Experiments aiming at dissecting the molecular basis of local adaptation must apprehend the complexity introduced by temporal fluctuations because they massively affect the expression of phenotype and fitness differences.