Abstract. Tropical forest dynamics play crucial roles in the global carbon, water, and energy cycles. Dynamic global vegetation models are the primary tools to simulate terrestrial ecosystem dynamics and their response to climate change. However, realistically simulating the dynamics of competition and coexistence of differing plant functional traits within tropical forests remains a significant challenge. This study aims to improve modeling of plant functional type (PFT) coexistence in the Functionally Assembled Terrestrial Ecosystem Simulator (FATES), a vegetation demography model implemented in the Energy Exascale Earth System Model (E3SM) land model (ELM), ELM-FATES. Specifically, we explore: (1) whether plant trait relationships established from field measurements can constrain ELM-FATES simulations; and (2) whether machine learning based surrogate models can emulate the complex ELM-FATES model and optimize parameter selections to improve PFT coexistence modeling. We conducted ELM-FATES experiments for a tropical forest site near Manaus, Brazil. We first conducted two ensembles of ELM-FATES experiments, without (Exp-1) and with (Exp-2) consideration of observed trait relationships, respectively. Considering the observed trait relationships (Exp-2) slightly improves ELM-FATES simulations of water, energy, and carbon fluxes, but degrades the simulation of PFT coexistence. Using eXtreme Gradient Boosting (XGBoost) based surrogate models trained on Exp-1, we optimize the trait-related parameters in ELM-FATES to enable PFT coexistence and reduce model errors relative to the field observations. We used parameters selected by the surrogate model to conduct another ensemble of ELM-FATES experiments (Exp-3). The probability of experiments yielding PFT coexistence greatly increases from 21 % in Exp-1 to 73 % in Exp-3. Further filtering those experiments that allow for PFT coexistence to agree within 15 % of the observations, Exp-3 still has 33 % of experiments left, much higher than the 1.4 % in Exp-1. Exp-3 also better reproduces the annual means and seasonal variations of water, energy and carbon fluxes, and the field inventory of above ground biomass. Our study demonstrates the benefits of using machine learning models to improve PFT coexistence modeling in ELM-FATES, with important implications for modeling the response and feedback of ecosystem dynamics to climate change. Our results also suggest that new mechanisms are required for robust simulation of coexisting plants in FATES.