Traditional knowledge is influenced by ancestry, inter-cultural diffusion and interaction with the natural environment. It is problematic to assess the contributions of these influences independently because closely related ethnic groups may also be geographically close, exposed to similar environments and able to exchange knowledge readily. Medicinal plant use is one of the most important components of traditional knowledge, since plants provide healthcare for up to 80% of the world's population. Here, we assess the significance of ancestry, geographical proximity of cultures and the environment in determining medicinal plant use for 12 ethnic groups in Nepal. Incorporating phylogenetic information to account for plant evolutionary relatedness, we calculate pairwise distances that describe differences in the ethnic groups' medicinal floras and floristic environments. We also determine linguistic relatedness and geographical separation for all pairs of ethnic groups. We show that medicinal uses are most similar when cultures are found in similar floristic environments. The correlation between medicinal flora and floristic environment was positive and strongly significant, in contrast to the effects of shared ancestry and geographical proximity. These findings demonstrate the importance of adaptation to local environments, even at small spatial scale, in shaping traditional knowledge during human cultural evolution.