Species of the highly diverse fungal genus Aspergillus are well-known agricultural pests, and, most importantly, producers of various mycotoxins threatening food safety worldwide. Mycotoxins are studied predominantly from the perspectives of human and livestock health. Meanwhile, their roles are far less known in nature. However, to understand the factors behind mycotoxin production, the roles of the toxins of Aspergilli must be understood from a complex ecological perspective, taking mold-plant, mold-microbe, and mold-animal interactions into account. The Aspergilli may switch between saprophytic and pathogenic lifestyles, and the production of secondary metabolites, such as mycotoxins, may vary according to these fungal ways of life. Recent studies highlighted the complex ecological network of soil microbiotas determining the niches that Aspergilli can fill in. Interactions with the soil microbiota and soil macro-organisms determine the role of secondary metabolite production to a great extent. While, upon infection of plants, metabolic communication including fungal secondary metabolites like aflatoxins, gliotoxin, patulin, cyclopiazonic acid, and ochratoxin, influences the fate of both the invader and the host. In this review, the role of mycotoxin producing Aspergillus species and their interactions in the ecosystem are discussed. We intend to highlight the complexity of the roles of the main toxic secondary metabolites as well as their fate in natural environments and agriculture, a field that still has important knowledge gaps.