A goal of modern biology is to develop the genotype-phenotype (G-P) map, a predictive understanding of how genomic information generates the organismal trait variation that forms the basis of both natural and managed communities. As microbiome research advances, however, it has become clear that many of these traits are governed by genetic variation encoded not only by the host's own genome, but also by the genomes of myriad cryptic symbionts. Thus many ecologically-important traits are likely symbiotic extended phenotypes, and this recognition adds even more complexity to our conceptions of the G-P map. In model symbioses such as the legume-rhizobium mutualism, host growth and fitness often depend on genetic variation in symbiont partner quality, and our ability to manipulate host and symbiont genotype combinations, combined with increasingly precise sequencing and mapping approaches, provides an opportunity to characterize the genetic nature of these symbiotic extended phenotypes. Here we use naturally-occurring genetic variation in 191 strains of the nitrogen-fixing symbiont Ensifer meliloti in four mapping experiments to study the genomic architecture of symbiotic partner quality within and across environmental contexts and host genotypes. We demonstrate the quantitative genetic nature of symbiotic extended phenotypes, including extensive context-dependency in both the identity and functions of symbiont loci that control host growth. We additionally resolve a core set of universal loci from populations in the native range that are likely important in all or most environments, and thus, serve as excellent targets both for genetic engineering and future coevolutionary studies of symbiosis.