Statistically Consistent Rooting of Species Trees under the Multispecies Coalescent Model

Abstract

Rooted species trees are used in several downstream applications of phylogenetics. Most species tree estimation methods produce unrooted trees and additional methods are then used to root these unrooted trees. Recently, Quintet Rooting (QR) (Tabatabaee et al., ISMB and Bioinformatics 2022), a polynomial-time method for rooting an unrooted species tree given unrooted gene trees, was introduced. QR, which is based on a proof of identifiability of rooted 5-taxon trees in the presence of incomplete lineage sorting, was shown to have good accuracy, improving over other methods for rooting species trees when incomplete lineage sorting was the only cause of gene tree discordance, except when gene tree estimation error was very high. However, that study left the statistical consistency of QR as an open question. We present QR-STAR, a polynomial-time variant of QR that has an additional step for determining the rooted shape of each quintet tree. We prove that QR-STAR is statistically consistent under the multi-species coalescent (MSC) model. Our simulation study under a variety of model conditions shows that QR-STAR matches or improves on the accuracy of QR. QR-STAR is available in open source form at https://github.com/ytabatabaee/Quintet-Rooting.

DISCO+QR: rooting species trees in the presence of GDL and ILS

“…We also note that modifications to the DISCO+QR pipeline [e.g. changes to the QR optimization problem, such as using QR-STAR, an improved variant of QR ( Tabatabaee et al , 2022b )] may improve its accuracy. This study mainly examined small numbers of species, yet DISCO+QR clearly improves significantly in accuracy on larger numbers of species; hence, further study of conditions with more species is necessary.…”