Some of the most common ecological interactions are between plants and herbivorous insects, and these relationships are central to the study of ecological specialization. We address established assumptions about the positive association between local abundance and dietary specialization using a 17-year dataset on the caterpillars of Ecuador. Our long-term data include standardized plot-based samples as well as general, regional collections, allowing for investigations across spatial scales and using different indices of abundance for 1917 morphospecies of Lepidoptera from 33 families. We find that specialists are locally more abundant than generalists, consistent with a key component of the “jack of all trades, master of none” hypothesis, which has otherwise received poor to mixed support from previous studies that have mostly involved fewer species and shorter time series. Generalists achieve greater prevalence across the landscape, and we find some evidence for geographic variation in the abundance-diet breadth relationship, in particular among elevational bands. Interspecific variation in abundance also had a negative relationship with diet breadth, with specialists having more variable abundances across species. The fact that more specialized species can be both rare and common highlights the ecological complexity of specialization.