The Effect of Gene Flow on Coalescent-based Species-Tree Inference

Abstract

Most current methods for inferring species-level phylogenies under the coalescent model assume that no gene flow occurs following speciation. Several studies have examined the impact of gene flow (e.g., Eckert and Carstens 2008; Chung and Ané 2011; Leaché et al. 2014; Solís-Lemus et al. 2016) and of ancestral population structure (DeGeorgio and Rosenberg 2016) on the performance of species-level phylogenetic inference, and analytic results have been proven for network models of gene flow (e.g., Solís-Lemus et al. 2016; Zhu et al. 2016). However, there are few analytic results for a continuous model of gene flow following speciation, despite the development of mathematical tools that could facilitate such study (e.g., Hobolth et al. 2011; Andersen et al. 2014; Tian and Kubatko 2016). In this article, we consider a three-taxon isolation-with-migration model that allows gene flow between sister taxa for a brief period following speciation, as well as variation in the effective population sizes across the species tree. We derive the probabilities of each of the three gene tree topologies under this model, and show that for certain choices of the gene flow and effective population size parameters, anomalous gene trees (i.e., gene trees that are discordant with the species tree but that have higher probability than the gene tree concordant with the species tree) exist. We characterize the region of parameter space producing anomalous trees and show that the probability of the gene tree that is concordant with the species tree can be arbitrarily small. We then show that there is theoretical support for using SVDQuartets with an outgroup to infer the rooted three-taxon species tree in a model of gene flow between sister taxa. We study the performance of SVDQuartets on simulated data and compare it to three other commonly-used methods for species tree inference, ASTRAL, MP-EST, and concatenation. The simulations show that ASTRAL, MP-EST, and concatenation can be statistically inconsistent when gene flow is present, while SVDQuartets performs well, though large sample sizes may be required for certain parameter choices.

Phylogenomic analyses reveal a deep history of hybridization and polyploidy in the Neotropical genus Lachemilla (Rosaceae)

“…The varying placements of the Orbiculate clade when analyzing different datasets and/or using different approaches to estimate the species tree seems to be primarily the product of the inconsistency of species tree estimation in the presence of gene flow (Solís‐Lemus et al ., ; Long & Kubatko, ). Our network analysis of the HYBRID‐REDUCED dataset using P hylo N et revealed that all models involving reticulation events fit our data better than any model with strict bifurcating trees (Table ).…”

“…However, when investigating the evolutionary history of species, typically, strictly bifurcating species tree methods that account only for ILS are applied, and the potential impact of gene flow is not taken in account during the inference process. Moreover, mounting evidence that species tree methods are inconsistent in the presence of gene flow (Solís‐Lemus et al ., ; Long & Kubatko, ) demonstrates the need to incorporate methods that account for ILS and gene flow simultaneously in phylogenetic studies. Here, we present a clear example of the utility of these methods to clarify the evolutionary history of Lachemilla .…”

“…Differences between inferred trees from species tree and concatenation methods have been explained by the presence of ILS (Warnow, ) or gene tree estimation error (Springer & Gatesy, ). However, recent studies (Solís‐Lemus et al ., ; Long & Kubatko, ) revealed that species tree and concatenation methods can also be inconsistent in the presence of gene flow. Moreover, Solís‐Lemus et al .…”

The Impact of Cross-Species Gene Flow on Species Tree Estimation

“…3) calculated gene-tree probabilities under an introgression model, arguing that the concept of a species tree is poorly defined when there is gene flow. Long and Kubatko (2018) studied the probabilities of gene tree topologies under an isolation-with-initial-migration model (Wilkinson-Herbots, 2012) for three species, in which there is initial gene flow between sister species after their divergence. One might expect gene flow between sister species to make them appear more similar, making it easier to infer the species tree, but surprisingly with different population sizes, gene flow between sister species can cause the most probable gene tree topology to differ from the species tree, leading to so-called anomalous gene trees.…”

Stream flow alone does not predict population structure of diving beetles across complex tropical landscapes

“…Therefore, SVDquartets allows the consideration of both mutational and coalescent variance in the estimation of phylogenetic relationships (Chifman & Kubatko, 2014). SVDquartets has also been shown to perform better than other species tree inference programs in presence of gene flow (Long & Kubatko, 2018). SVDquartets assumes that each site has its own genealogy drawn from the coalescent model (Chifman & Kubatko, 2014).…”