The acquisition of new hosts provides a virus with more opportunities for transmission and survival but may be limited by across-host fitness trade-offs. Major causes of across-host trade-offs are antagonistic pleiotropy, that is, host differential phenotypic effects of mutations, a Genotype x Environment interaction, and epistasis, a Genotype x Genotype interaction. Here, we analyze if there are trade-offs, and what are the causes, associated with the acquisition by tobacco mild green mosaic virus (TMGMV) of a new host. For this, the multiplication of sympatric field isolates of TMGMV from its wild reservoir host Nicotiana glauca and from pepper crops was quantified in the original and the heterologous hosts. TMGMV isolates from N. glauca were adapted to their host, but pepper isolates were not adapted to pepper, and the acquisition of this new host was associated with a fitness penalty in the original host. Analyses of the collection of field isolates and of mutant genotypes derived from biologically active cDNA clones showed a role of mutations in the coat protein and the 3′ untranslated region in determining within-host virus fitness. Fitness depended on host-specific effects of these mutations, on the genetic background in which they occurred, and on higher-order interactions of the type Genotype x Genotype x Environment. These types of effects had been reported to generate across-host fitness trade-offs under experimental evolution. Our results show they may also operate in heterogeneous natural environments and could explain why pepper isolates were not adapted to pepper and their lower fitness in N. glauca.IMPORTANCE The acquisition of new hosts conditions virus epidemiology and emergence; hence it is important to understand the mechanisms behind host range expansion. Experimental evolution studies have identified antagonistic pleiotropy and epistasis as genetic mechanisms that limit host range expansion, but studies from virus field populations are few. Here, we compare the performance of isolates of tobacco mild green mosaic virus from its reservoir host, Nicotiana glauca, and its new host, pepper, showing that acquisition of a new host was not followed by adaptation to it but was associated with a fitness loss in the original host. Analysis of mutations determining host-specific virus multiplication identified antagonistic pleiotropy, epistasis, and host-specific epistasis as mechanisms generating across-host fitness trade-offs that may prevent adaptation to pepper and cause a loss of fitness in N. glauca. Thus, mechanisms determining trade-offs, identified under experimental evolution, could also operate in the heterogeneous environment in which natural plant virus populations occur.