Ancestral Population Genomics: The Coalescent Hidden Markov Model Approach

Abstract

With incomplete lineage sorting (ILS), the genealogy of closely related species differs along their genomes. The amount of ILS depends on population parameters such as the ancestral effective population sizes and the recombination rate, but also on the number of generations between speciation events. We use a hidden Markov model parameterized according to coalescent theory to infer the genealogy along a fourspecies genome alignment of closely related species and estimate population parameters. We analyze a basic, panmictic demographic model and study its properties using an extensive set of coalescent simulations. We assess the effect of the model assumptions and demonstrate that the Markov property provides a good approximation to the ancestral recombination graph. Using a too restricted set of possible genealogies, necessary to reduce the computational load, can bias parameter estimates. We propose a simple correction for this bias and suggest directions for future extensions of the model. We show that the patterns of ILS along a sequence alignment can be recovered efficiently together with the ancestral recombination rate. Finally, we introduce an extension of the basic model that allows for mutation rate heterogeneity and reanalyze human-chimpanzee-gorilla-orangutan alignments, using the new models. We expect that this framework will prove useful for population genomics and provide exciting insights into genome evolution. B IOLOGICAL sequence data, and particularly the variation therein, contain information about the evolutionary processes that shaped the present-day organisms. Coalescent theory provides tools for comparative sequence analysis to investigate the history of populations, by studying the genealogy of the sampled sequences (Hein et al. 2005). More recently, with the rapid accumulation of molecular data, multiple-loci studies have become possible, allowing for the estimation of population genetics parameters such as speciation times and ancestral population sizes (Rannala and Yang 2003; Burgess and Yang 2008). The availability of complete genome sequences for closely related species opens a new area of research, by providing virtually as many loci as possible, yet for a single sequence from a limited number of species. While such data cannot be used to study contemporary populations, they contain information about ancestral population processes, particularly when speciation events are sufficiently close in time that incomplete lineage sorting (ILS) occurs.Consider a site in an alignment of human, chimpanzee, and gorilla. The most likely evolutionary scenario is that going backward in time, the human and chimpanzee sequences coalesce first within the humanchimpanzee (HC) ancestral population and then meet the gorilla sequence within the human-chimpanzeegorilla (HCG) ancestral population (Figure 1, case HC1). Because of genetic drift, we expect the sequences to have an older common ancestor in some regions, falling back in the HCG ancestral population. The two lineages would have been passe...

Incomplete lineage sorting patterns among human, chimpanzee, and orangutan suggest recent orangutan speciation and widespread selection

“…The existence of ILS with orangutan provides an alternative estimate of the speciation time between human and chimpanzee. The result of 4.2 Mya is consistent with recent analyses using a part of the gorilla genome (Patterson et al 2006;Burgess and Yang 2008;Dutheil et al 2009). Such a recent speciation event suggests that Ardipithecus ramidus may predate the human-chimpanzee speciation event (TD ).…”

“…CoalHMM analysis Dutheil et al (2009) describe a model that allows for genealogies to change along an alignment of three species and an outgroup. We apply it to a genome-wide HCOM (human, chimpanzee, and orangutan with macaque as outgroup) alignment.…”

“…We use the patterns of ILS to disentangle human-orangutan speciation from humanorangutan divergence. This allows us to estimate the population sizes of the ancestral species (e.g., Chen and Li 2001;Hobolth et al 2007;Dutheil et al 2009) as well as the variations in the effective population sizes for different parts of the genome (e.g., McVicker et al 2009). More than 100,000 very short segments of the genome support gene genealogies different from the species tree.…”

Estimating Variable Effective Population Sizes from Multiple Genomes: A Sequentially Markov Conditional Sampling Distribution Approach

“…In a related line of research, interesting progress has been made (Hobolth et al 2007;Dutheil et al 2009;Mailund et al 2011) in performing "ancestral population genomic" inference under a coalescent HMM, but its applicability is limited to only a modest number of sequences, again due to the explosion in the state space.…”

Using Colored Petri Nets to Construct Coalescent Hidden Markov Models: Automatic Translation from Demographic Specifications to Efficient Inference Methods

“…Sequential Markov Coalescent (SMC) and its inference method Coalescent hidden Markov models (CoalHMMs) [5,8,9,12,18,22,24] is a recently developed methodology for analyzing genome relationships and make inference of speciation divergence and the mechanisms involved in speciation. CoalHMMs combine the so-called "coalescence process" model of population genetics [11] with the computational efficient statistical tool "hidden Markov models" [7] and provides the first approach to analyze the speciation process computationally scalable to whole-genome analysis.…”