The conditional mutualism between scatterhoarders and trees varies on a continuum from mutualism to antagonism and can change across time and space, and among species. We examined 4 tree species (red oak [Quercus rubra], white oak [Quercus alba], American chestnut [Castanea dentata] and hybrid chestnut [C. dentata × Castanea mollissima) across 5 sites and 3 years to quantify the variability in this conditional mutualism. We used a published model to compare the rates of seed emergence with and without burial to the probability that seeds will be cached and left uneaten by scatterhoarders to quantify variation in the conditional mutualism that can be explained by environmental variation among sites, years, species, and seed provenance within species. All species tested had increased emergence when buried. However, comparing benefits of burial to the probability of caching by scatterhoarders indicated a mutualism in red oak, while white oak was nearly always antagonistic. Chestnut was variable around the boundary between mutualism and antagonism, indicating a high degree of context dependence in the relationship with scatterhoarders. We found that different seed provenances did not vary in their potential for mutualism. Temperature did not explain microsite differences in seed emergence in any of the species tested. In hybrid chestnut only, emergence on the surface declined with soil moisture in the fall. By quantifying the variation in the conditional mutualism that was not caused by changes in scatterhoarder behavior, we show that environmental conditions and seed traits are an important and underappreciated component of the variation in the relationship between trees and scatterhoarders.