Selection exerted by herbivores is a major force driving the evolution of plant defensive characters such as leaf trichomes or secondary metabolites. However, plant defense expression is highly variable among populations and identifying the sources of this variation remains a major challenge. Plant populations are often distributed across broad geographic ranges and are exposed to different herbivore communities, ranging from generalists (that feed on diverse plant species) to specialists (that feed on a restricted group of plants). We studied eight populations of the plant Datura stramonium usually eaten by specialist or generalist herbivores, in order to examine whether the pattern of phenotypic selection on secondary compounds (atropine and scopolamine) and a physical defense (trichome density) can explain geographic variation in these traits. Following co-evolutionary theory, we evaluated whether a more derived alkaloid (scopolamine) confers higher fitness benefits than its precursor (atropine), and whether this effect differs between specialist and generalist herbivores. Our results showed consistent directional selection in almost all populations and herbivores to reduce the concentration of atropine. The most derived alkaloid (scopolamine) was favored in only one of the populations, which is dominated by a generalist herbivore. In general, the patterns of selection support the existence of a selection mosaic and accounts for the positive correlation observed between atropine concentration and plant damage by herbivores recorded in previous studies.