To describe a living organism it is often said that “the whole is greater than the sum of its parts”. In genetics, we may also think that the effect of multiple mutations on an organism is greater than their additive individual effect, a phenomenon called epistasis or multiplicity. Despite the last decade’s discovery that many disease- and fitness-related traits are polygenic, or controlled by many genetic variants, it is still debated whether the effects of individual genes combine additively or not. Here we develop a flexible likelihood framework for genome-wide associations to fit complex traits such as fitness under both additive and non-additive polygenic architectures. Analyses of simulated datasets under different true additive, multiplicative, or other epistatic models, confirm that our method can identify global non-additive selection. Applying the model to experimental datasets of wild type lines of Arabidopsis thaliana, Drosophila melanogaster, and Saccharomyces cerevisiae, we find that fitness is often best explained with non-additive polygenic models. Instead, a multiplicative polygenic model appears to better explain fitness in some experimental environments. The statistical models presented here have the potential to improve prediction of phenotypes, such as disease susceptibility, over the standard methods for calculating polygenic scores which assume additivity.