Correlation of coalescence times in a diploid Wright-Fisher model with recombination and selfing

Abstract

The correlation among the gene genealogies at different loci is crucial in biology, yet challenging to understand because such correlation depends on many factors including genetic linkage, recombination, natural selection and population structure. Based on a diploid Wright-Fisher model with a single mating type and partial selfing for a constant large population with size N, we quantify the combined effect of genetic drift and two competing factors, recombination and selfing, on the correlation of coalescence times at two linked loci for samples of size two. Recombination decouples the genealogies at different loci and decreases the correlation while selfing increases the correlation. We obtain explicit asymptotic formulas for the correlation for four scaling scenarios that depend on whether the selfing probability and the recombination probability are of order O(1/N) or O(1) as N tends to infinity. Our analytical results confirm that the asymptotic lower bound in [King, Wakeley, Carmi (TPB 2018)] is sharp when the loci are unlinked and when there is no selfing, and provide a number of new formulas for other scaling scenarios that have not been considered before. We present asymptotic results for the variance of Tajima's estimator of the population mutation rate for infinitely many loci as N tends to infinity. When the selfing probability is of order O(1) and is equal to a positive constant s for all $N$ and if the samples at both loci are in the same individual, then the variance of the Tajima's estimator tends to s/2 (hence remains positive) even when the recombination rate, the number of loci and the population size all tend to infinity. We present asymptotic results for the variance of Tajima's estimator of the population mutation rate for infinitely many loci as N tends to infinity. When the selfing probability is of order O(1) and is equal to a positive constant s for all N and if the samples at both loci are in the same individual, then the variance of the Tajima's estimator tends to s/2 (hence remains positive) even when the recombination rate, the number of loci and the population size all tend to infinity.