Communities of herbivorous insects on individual host trees may be driven by processes ranging from ongoing development via recent microevolution to ancient phylogeny, but the relative importance of these processes and whether they operate via trophic interactions or herbivore movement remains unknown. We determined the leaf phenology, trunk diameter, genotype, and neighbourhood of sessile oak trees (Quercus petraea), and sampled their caterpillar communities. We found that leaf development across a time period of days related to free-living caterpillars, which disappeared with leaf age. Tree growth across decades is related to increased parasitism rate and diversity of herbivores. The microevolution of oak trees across millennia is related to the abundance of leaf-mining casebearers, which is higher on more homozygous oaks. However, oak genome size was not important for any guild. In contrast to most previous studies, the phylogenetic distance of oaks from their neighbours measured in millions of years was associated with higher abundances of entire caterpillar guilds. Furthermore, on trees surrounded by only distantly related tree species, parasitism tended to be lower. Lower parasitism, in turn, was associated with higher abundances of codominant caterpillar species. Neighbourhoods and traits of trees were also related to community composition and diversity, but not to the average wingspans or specialization of species, consistent with the assembly of herbivore communities being driven by leaf traits and parasitism pressure on trees rather than by insect movement among trees. However, movement in rarer species may be responsible for concentration effects in more phylogenetically distant neighbourhoods. Overall, we suggest that the assembly of insects on a tree is mostly driven by trophic interactions controlled by a mosaic of processes playing out over very different time scales. Comparisons with the literature further suggest that, for oak trees, the consequences of growing amongst distantly related tree species may depend on factors such as geographic region and tree age.