Yields are in decline for a number of crops and regions, but the drivers of this process are poorly known. Global decreases in pollinator abundance and diversity have been proposed as a major driver of yield declines in crops that depend on animals, mostly bees, to produce fruits and seeds. Alternatively, widespread tree mortality has been directly and indirectly related to global climate change, which could also explain yield decreases in tree crops. As tree crops are expected to be more dependent on pollinators than other crop types, disentangling the relative influence of growth form and pollinator dependence is relevant to identify the ultimate factors driving yield declines. The incidence of yield decline, estimated as a negative average yield growth rate from 1961 to 2020, was measured in 4270 time series, involving 136 crops and 163 countries and territories. About one-fourth of all time series showed negative average rates of yield growth, a characteristic associated with both high pollinator dependence and a tree growth form. Because pollinator dependence and plant growth form were partially correlated, we disentangled the effect of each of these two predictors using a series of generalized linear mixed models that evaluated direct and indirect associations. Our analyses revealed a stronger association of yield decline with growth form than with pollinator dependence, a relationship that persisted after partialling out the effect of pollinator dependence. In particular, yield declines were more common among tree than herbaceous and shrub crops in all major regions but in Africa, a continent showing a high incidence of yield declines irrespective of growth form. These results suggest that global climate change rather than pollinator decline could already be impacting tree crop productivity.