As one of the biggest agroproducts producers, China plays an important role in the global supply. Yet, climate change inevitably threatens their production and leads to tremendous losses. Furthermore, climatic and nonclimatic factors are likely to influence their producing behaviors and yields. Accordingly, this work aims to explore both the qualitative and quantitative nexus between climate change, nonclimatic drivers, and agroproduct yield in China. We choose lychee (Litchi chinensis Sonn.), with world's largest production in China and one of the most demanding agroproduct for climatic conditions, as the subject. A two-way fixed-effect Poisson model with robust standard error is developed based on county-level panel data in 39 main producing counties in China along with climatic and cultural festival factors from 2014 to 2019. The main conclusions are as follows: (a) the lychee culture festival, a nonclimatic driver, has negative effect on lychee yield, and this is a novel effect pathway theoretically and we verify it empirically; (b) climate changes in various phenological phases are significantly correlated to lychee yield; precipitation during the exposure phase or flowering phase has negative effect, and minimum temperature during the heading phase has positive effect, which is the first paper in this field; and (c) a new method is developed to analyze nonnegative yield and production, which could also be applied in other industries.