Coffee, an important global commodity, is threatened by climate change. Agroforestry has been considered as one option to maintain or enhance coffee production. In this study, we use a machine learning ensemble consisting of MaxEnt, Random Forest and Boosted Regression Trees to assess climate change impacts on the suitability to grow Arabica coffee, Robusta coffee and bananas in Uganda by 2050. Based on this, the buffering potential of Cordia africana and Ficus natalensis, the two commonly used shading trees in agroforestry systems is assessed. Our robust models (AUC of 0.7–0.9) indicate temperature-related variables as relevant for Arabica coffee suitability, while precipitation-related variables determine Robusta coffee and banana suitability. Under current climatic conditions, only a quarter of the total land area is suitable for growing Arabica coffee, while over three-quarters are suitable for Robusta coffee and bananas. Our results suggest that climate change will reduce the area suitable to grow Arabica coffee, Robusta coffee and bananas by 20%, 9% and 3.5%, respectively, under SSP3-RCP7.0 by 2050. A shift in areas suitable for Arabica coffee to highlands might occur, leading to potential encroachment on protected areas. In our model, implementing agroforestry with up to 50% shading could partially offset suitable area losses for Robusta coffee—but not for Arabica coffee. The potential to produce valuable Arabica coffee thus decreases under climate change and cannot be averted by agroforestry. We conclude that the implementation and design of agroforestry must be based on species, elevation, and regional climate projections to avoid maladaptation.