The niche and fitness differences of modern coexistence theory separate mechanisms into stabilizing and equalizing components. Although this decomposition can help us predict and understand species coexistence, the extent to which mechanistic inference is sensitive to the method used to partition niche and fitness differences remains unclear. We apply two alternative methods to assess niche and fitness differences to four well known community models. We show that because standard methods based on linear approximations do not capture the full community dynamics, they can sometimes lead to incorrect predictions of coexistence and misleading interpretations of stabilizing and equalizing mechanisms. Conversely, a more recently developed method to decompose niche and fitness differences, that accounts for the full nonlinear dynamics of competition, consistently identifies the correct contribution of stabilizing and equalizing components. This approach further reveals that when the true complexity of the system is taken into account, essentially all mechanisms comprise both stabilizing and equalizing components. Amidst growing interest in the role of non-additive and higher-order interactions in regulating species coexistence, we propose that the effective decomposition of niche and fitness differences will become increasingly reliant on methods that account for the inherent non-linearity of community dynamics.

Trait diversity is traditionally seen as promoting species richness and ecosystem function. Species with dissimilar traits would partition available resources, increasing niche differences, facilitating coexistence and increasing ecosystem function. Here we first show, using theory and simulations for light-limited phytoplankton, that combing photosynthetic pigments is indeed a necessary condition for coexistence and stimulates ecosystem function. However, pigment richness does mostly not permit the coexistence of more than two species, and increases productivity at most 60% compared to single-pigment communities. Surprisingly, combining all nine pigments known to date leads to a 2.5% probability that four species would coexist, illustrating that the coexistence of a high number of species along a continuous niche axis is constrained by limiting similarity. We explain these constraints by unimodal effects of pigment richness on niche and fitness differences, which jointly limit the positive effect of pigment on species richness. Empirical data and additional simulations suggest that pigment richness effects can be stronger during transient dynamics but inevitably weaken with time, i.e. pigment richness effects on species richness and function are likely short-lived. Our results highlight the need to apply coexistence theory to understand the long-term effects of trait diversity on biodiversity and ecosystem function.Statement of authorshipJ.W.S. and F.dL. developed the ideas and wrote the manuscript. J.W.S developed the mathematics and the python code to conduct the study. J.W.S conducted the literature review.