Plants are widely recognized as chemical factories, with each species producing dozens to hundreds of unique secondary metabolites. These compounds shape the interactions between plants and their natural enemies. Here we explore how plants generate chemical diversity, and what evolutionary processes have led to novel compounds and unique chemical profiles. We comprehensively characterized the chemical profile of one-third of the species of tropical rainforest trees in the genus Inga (~ 100, Fabaceae) and applied phylogenetic comparative methods to understand the mode of chemical defense evolution. We show that: 1) Each Inga species produces exceptionally high levels of phytochemical diversity, despite costs, tradeoffs and biosynthetic constraints. 2) Closely related species have highly divergent defense profiles, with individual compounds, major compound classes and complete profiles showing little to no phylogenetic signal. 3) We show that the evolution of a species' chemical profile shows a signature of divergent adaptation, implying that it is advantageous for a species to have distinct chemistry from close relatives to avoid shared natural enemies. 4) Finally, we hypothesize a model where deep homology of biosynthetic pathways and rapid changes in regulatory mechanisms may better explain the observed large shifts in defense chemicals between closely related taxa.