One of the fundamental questions in biology is how the genotype regulates the phenotype. An increasing number of studies indicate that, in most cases, the effect of a genetic locus on the phenotype is context-dependent, i.e. it is influenced by the genetic background and the environment in which the phenotype is measured. Still, the majority of the studies, in both model organisms and humans, that map the genetic regulation of phenotypic variation in complex traits primarily identify additive loci with independent effects. This does not reflect an absence of the contribution of genetic interactions to phenotypic variation, but instead is a consequence of the technical limitations in mapping gene-gene interactions (GGI) and gene-environment interactions (GEI). Yeast, with its detailed molecular understanding, diverse population genomics and ease of genetic manipulation, is a unique and powerful resource to study the contributions of GGI and GEI in the regulation of phenotypic variation. Here we review studies in yeast that have identified GGI and GEI that regulate phenotypic variation, and discuss the contribution of these findings in explaining missing heritability of complex traits, and how observations from these GGI and GEI studies enhance our understanding of the mechanisms underlying genetic robustness and adaptability that shape the architecture of the genotype-phenotype map.