Two-locus two-allele models are among the most studied models in population genetics. The reason is that they are the simplest models to explore the role of epistasis for a variety of important evolutionary problems, including the maintenance of polymorphism and the evolution of genetic incompatibilities. Many specific types of models have been explored. However, due to the mathematical complexity arising from the fact that epistasis generates linkage disequilibrium, few general insights have emerged. Here, we study a simpler problem by assuming that linkage disequilibrium can be ignored. This is a valid approximation if selection is sufficiently weak relative to recombination. The goal of our paper is to characterize all possible equilibrium structures, or more precisely and general, all robust phase portraits or evolutionary flows arising from this weak-selection dynamics. For general fitness matrices, we have not fully accomplished this goal, because some cases remain undecided. However, for many specific classes of fitness schemes, including additive fitnesses, purely additive-by-additive epistasis, haploid selection, multilinear epistasis, marginal overdominance or underdominance, and the symmetric viability model, we obtain complete characterizations of the possible equilibrium structures and, in several cases, even of all possible phase portraits. A central point in our analysis is the inference of the number and stability of fully polymorphic equilibria from the boundary flow, i.e., from the dynamics at the four marginal single-locus subsystems. The key mathematical ingredient for this is index theory. The specific form of epistasis has both a big influence on the possible boundary flows as well as on the internal equilibrium structure admitted by a given boundary flow.Electronic supplementary materialThe online version of this article (doi:10.1007/s00285-017-1140-7) contains supplementary material, which is available to authorized users.

No abstract

We study local adaptation of a peripheral population by investigating the fate of new mutations using a haploid two-locus two-allele continent-island migration model. We explore how linkage, epistasis, and maladaptive gene flow affect the invasion probability of weakly beneficial de-novo mutations that arise on the island at an arbitrary physical distance to a locus that already maintains a stable migration-selection polymorphism. By assuming a slightly supercritical branching process, we deduce explicit conditions on the parameters that permit a positive invasion probability and we derive approximations for it. They show how the invasion probability depends on the additive and epistatic effects of the mutant, on its linkage to the polymorphism, and on the migration rate. We use these approximations together with empirically motivated distributions of epistatic effects to analyze the influence of epistasis on the expected invasion probability if mutants are drawn randomly from such a distribution and occur at a random physical distance to the existing polymorphism. We find that the invasion probability generally increases as the epistasis parameter increases or the migration rate decreases, but not necessarily as the recombination rate decreases. Finally, we shed light on the size of emerging genomic islands of divergence by exploring the size of the chromosomal neighborhood of the already established polymorphism in which 50% or 90% of the successfully invading mutations become established. These 'window sizes' always decrease in a reverse sigmoidal way with stronger migration and typically increase with increasing epistatic effect.

In the evolutionary biology literature, it is generally assumed that in deterministic haploid selection models, in the absence of variation-generating mechanisms such as mutation, no polymorphic equilibrium can be stable. However, results corroborating this claim are scarce and almost always depend upon additional assumptions. Using ideas from game theory, we establish a condition on the fitness parameters of haplotypes formed by two loci such that a monomorphism is a global attractor. Further, we show that no isolated equilibrium exists, at which an unequal number of alleles from two loci is present. Under the assumption of convergence of trajectories to equilirium points, we settle the two-locus three-allele case for a fitness scheme formally equivalent to the classical symmetric viability model.