Diminishing returns epistasis causes the benefit of the same advantageous mutation smaller in fitter genotypes, and is frequently observed in experimental evolution. However, its occurrence in other contexts, environment-dependence, and mechanistic basis are unclear. Here we address these questions using 1005 sequenced segregants generated from a yeast cross.Under each of 47 examined environments, 66-92% of tested polymorphisms exhibit diminishing returns epistasis. Surprisingly, improving environment quality also reduces the benefits of advantageous mutations even when fitness is controlled for, indicating the necessity to revise the global epistasis hypothesis. We propose that diminishing returns originates from the modular organization of life where the contribution of each functional module to fitness is determined jointly by the genotype and environment and has an upper limit, and demonstrate that our model predictions match empirical observations. These findings broaden the concept of diminishing returns epistasis, reveal its generality and potential cause, and have important evolutionary implications.