Many organisms change their phenotype in response to the environment, a phenomenon called phenotypic plasticity. Although plasticity can dramatically change the phenotype of an organism, we hardly understand how this can affect biotic interactions and the resulting phenotypic selection. Here, we use fast cycling Brassica rapa plants in an experiment in the greenhouse to study the link between plasticity and selection. We detected strong plasticity in morphology, nectar, and floral scent in response to different soil types and aphid herbivory. We found positive selection on nectar and morphological traits in hand‐ and bumblebee‐pollinated plants. Bumblebee‐mediated selection on a principal component representing plant height, flower number, and flowering time (mPC3) differed depending on soil type and herbivory. For plants growing in richer soil, selection was stronger in the absence of herbivores, whereas for plants growing in poorer soil selection was stronger with herbivory. We showed that bumblebees visited tall plants with many flowers overproportionally in plants in poor soil with herbivory (i.e., when tall plants were rare), thus causing stronger positive selection on this trait combination. We suggest that with strong plasticity under most stressful conditions, pollinator‐mediated selection may promote adaptation to local environmental factors given sufficient heritability of the traits under selection.