Bray‐Curtis (AFD) differentiation in molecular ecology: Forecasting, an adjustment (^AA), and comparative performance in selection detection

Abstract

Geographic genetic differentiation measures are used for purposes such as assessing genetic diversity and connectivity, and searching for signals of selection. Confirmation by unrelated measures can minimize false positives. A popular differentiation measure, Bray‐Curtis, has been used increasingly in molecular ecology, renamed AFD (hereafter called BCAFD ). Critically, BCAFD is expected to be partially independent of the commonly used Hill “Q‐profile” measures. BCAFD needs scrutiny for potential biases, by examining limits on its value, and comparing simulations against expectations. BCAFD has two dependencies on within‐population (alpha) variation, undesirable for a between‐population (beta) measure. The first dependency is derived from similarity to and . The second dependency is that BCAFD cannot be larger than the highest allele proportion in either location (alpha variation), which can be overcome by data‐filtering or by a modified statistic A A or “Adjusted AFD”. The first dependency does not forestall applications such as assessing connectivity or selection, if we know the measure's null behavior under selective neutrality with specified conditions—which is shown in this article for A A , for equilibrium, and nonequilibrium, for the commonly used data type of single‐nucleotide‐polymorphisms (SNPs) in two locations. Thus, A A can be used in tandem with mathematically contrasting differentiation measures, with the aim of reducing false inferences. For detecting adaptive loci, the relative performance of A A and other measures was evaluated, showing that it is best to use two mathematically different measures simultaneously, and that A A is in one of the best such pairwise criteria. For any application, using A A , rather than BCAFD, avoids the counterintuitive limitation by maximum allele proportion within localities.

A Comparison of Entropic Diversity and Variance in the Study of Population Structure

“…The TDAP indices examined in this study all measure allelic differentiation, but they differ in the parameters that they measure: D′ AP tracks allelic diversity, the heterozygosity-based indices (G′ ST H, G″ ST (=F′ STv ), Jost’s D, D EST ) focus on the probability of alleles being in a heterozygous pairing, F′ STv measures variance among genotypes. Not a part of the Hill-number family of diversity measures [ 21 , 34 ], BCGD measures allelic differentiation by comparing the similarity of simple allele frequencies (or relative allele frequency) between two populations. Although BCGD was found by [ 21 ] to have close a relationship to G ST and F ST , this study unequivocally shows that BCGD yields results close to those of the q = 1-based SIDTA D′ AP and that they are highly divergent from those of the q = 2 indices (both expected heterozygosity based and variance based).…”

“…Not a part of the Hill-number family of diversity measures [ 21 , 34 ], BCGD measures allelic differentiation by comparing the similarity of simple allele frequencies (or relative allele frequency) between two populations. Although BCGD was found by [ 21 ] to have close a relationship to G ST and F ST , this study unequivocally shows that BCGD yields results close to those of the q = 1-based SIDTA D′ AP and that they are highly divergent from those of the q = 2 indices (both expected heterozygosity based and variance based).…”

“…An additional index, the Bray–Curtis index of dissimilarity [ 30 ] was also included. Originally formulated to assess differences between communities based on species importance values, this index has also been used to measure allelic differentiation between populations, using either an allele frequency format or a proportion format in place of species importance values (albeit with each format requiring different, but equivalent, equations) [ 20 , 21 ]. I will refer to this approach as Bray–Curtis genetic differentiation (BCGD).…”

“…They include: F STh , G ST , G′ ST N (Nei’s standardized G ST ), G′ ST H (Hedrick’s standardized G ST ), G″ ST (Hedrick’s standardized G ST , further corrected for bias when population number is small), Jost’s D, and D EST . Additionally, included is Allele Frequency Difference (AFD) [ 20 , 21 ], an approach based on either allele frequency or relative allele frequency data and hereafter referred to as Bray–Curtis Genetic Difference (BCGD).…”

Distances and their visualization in studies of spatial-temporal genetic variation using single nucleotide polymorphisms (SNPs)

“…The same issue has arisen in ecology (Yoshioka, 2008). Czekanowski Distance applied to SNP data and corrected for maximum value dependency is referred to as A A Distance (Sherwin, 2022). Chord Distance (Edwards & Cavalli-Sforza, 1964) (Table 2) assumes divergence between populations is via drift alone, and so again may be more appropriate than Nei's D for spatial population genetics.…”