Factors promoting the evolution of specialists versus generalists have been little studied in ecological context. In a large‐scale comparative field experiment, we studied genotypes from naturally evolved populations of a closely related generalist/specialist species pair (Polygonum persicaria and P. hydropiper), reciprocally transplanting replicates of multiple lines into open and partially shaded sites where the species naturally co‐occur. We measured relative fitness, individual plasticity, herbivory, and genetic variance expressed in the contrasting light habitats at both low and high densities. Fitness data confirmed that the putative specialist out‐performed the generalist in only one environment, the favorable full sun/low‐density environment to which it is largely restricted in nature, while the generalist had higher lifetime reproduction in both canopy and dense neighbor shade. The generalist, P. persicaria, also expressed greater adaptive plasticity for biomass allocation and leaf size in shaded conditions than the specialist. We found no evidence that the ecological specialization of P. hydropiper reflects either genetically based fitness trade‐offs or maintenance costs of plasticity, two types of genetic constraint often invoked to prevent the evolution of broadly adaptive genotypes. However, the patterns of fitness variance and herbivore damage revealed how release from herbivory in a new range can cause an introduced species to evolve as a specialist in that range, a surprising finding with important implications for invasion biology. Patterns of fitness variance between and within sites are also consistent with a possible role for the process of mutation accumulation (in this case, mutations affecting shade‐expressed phenotypes) in the evolution and/or maintenance of specialization in P. hydropiper.