Permutation tests for hypothesis testing with animal social network data: problems and potential solutions

Abstract

Generating insights about a null hypothesis requires not only a good dataset, but also statistical tests that are reliable and actually address the null hypothesis of interest. Recent studies have found that permutation tests, which are widely used to test hypotheses when working with animal social network data, can suffer from high rates of type I error (false positives) and type II error (false negatives). Here, we first outline why pre-network and node permutation tests have elevated type I and II error rates. We then propose a new procedure, the double permutation test, that addresses some of the limitations of existing approaches by combining pre-network and node permutations. We conduct a range of simulations, allowing us to estimate error rates under different scenarios, including errors caused by confounding effects of social or non-social structure in the raw data. We show that double permutation tests avoid elevated type I errors, while remaining sufficiently sensitive to avoid elevated type II errors. By contrast, the existing solutions we tested, including node permutations, pre-network permutations, and regression models with control variables, all exhibit elevated errors under at least one set of simulated conditions. Type I error rates from double permutation remain close to 5% in the same scenarios where type I error rates from pre-network permutation tests exceed 30%. The double permutation test provides a potential solution to issues arising from elevated type I and type II error rates when testing hypotheses with social network data. We also discuss other approaches, including restricted node permutations, testing multiple null hypotheses, and splitting large datasets to generate replicated networks, that can strengthen our ability to make robust inferences. Finally, we highlight ways that uncertainty can be explicitly considered during the analysis using permutation-based or Bayesian methods.

Oil Palm Cultivation Critically Affects Sociality in an Endangered Malaysian Primate

“…This included recalculating the network parameters after randomly swapping the nodes of the original networks. We used node-swapping (as opposed to generating random graphs or using pre-network 'edgeswapping' randomizations) since this approach seemed better suited for our purposes of testing regression-based null hypotheses in a taxon with a largely stable group composition and relatively low observation biases 44,45 . Speci cally, node-swapping preserves the overall size, number of connections, and structure of the network, thereby also preserving the overall distribution of node-level measures such as degree and EC.…”

“…Speci cally, node-swapping preserves the overall size, number of connections, and structure of the network, thereby also preserving the overall distribution of node-level measures such as degree and EC. It is therefore a more conservative approach that may be less susceptible to Type I errors, compared to random graph generation or edge-swapping 44,45 . After each permuted swapping event, we re t the same GLMM using these newly created parameters as response variable.…”

“…In contrast to partner diversity, EC signi cantly differed between habitats as indicated by the LRT (χ² = 55.01, df = 8, P < 0.001, N = 36 individuals, details in Supplementary Table S2). To account for interdependency of network measures, particularly EC, we additionally ran node-swapping permutation tests 44,45 which con rmed the signi cant effect of different habitats on EC (Supplementary Table S2).…”

Kinship does not predict the structure of a shark social network

“…Recent studies found that (datastream and node) permutation tests, which are widely used to test hypotheses in animal social network analyses, can produce high rates of type I error (false positives) and type II error (false negatives) (Franks et al 2020; Puga□Gonzalez et al 2020; Weiss et al 2020; Farine and Carter 2020). In order to ensure that the results provided by our GAI analyses are reliable and account for both social and non-social nuisance effects, we used an approach proposed by Farine and Carter (2020) that uses datastream permutations to control for nuisance effects, then uses node permutations to test for the statistical significance of the effect of interest. It first uses datastream permutations to calculate the deviation of each of a node-level or edge-level metric from its random expectation given the structure of the observation data (i.e.…”

“…This approach is similar to generalised affiliation indices (Whitehead and James 2015), but it uses datastream permutation tests, rather than regression models, to estimate the deviance from random. This approach demonstrated its robustness to test the role of kinship in shaping the strength of interactions between individuals in the presence of other social effects such as the presence of non-kin social bonds (Farine and Carter 2020). We therefore applied this double-permutation procedure to test for the role of kinship in driving associations among sharks as well as the role of sex on node centrality (controlling for the number of observations) and to confirm the robustness of our previous results to high type I and type II error rates.…”