A fast and unbiased procedure to randomize ecological binary matrices with fixed row and column totals

Abstract

A well-known problem in numerical ecology is how to recombine presence-absence matrices without altering row and column totals. A few solutions have been proposed, but all of them present some issues in terms of statistical robustness (that is, their capability to generate different matrix configurations with the same probability) and their performance (that is, the computational effort that they require to generate a null matrix). Here we introduce the 'Curveball algorithm', a new procedure that differs from existing methods in that it focuses rather on matrix information content than on matrix structure. We demonstrate that the algorithm can sample uniformly the set of all possible matrix configurations requiring a computational effort orders of magnitude lower than that required by available methods, making it possible to easily randomize matrices larger than 10 8 cells.

Proof of uniform sampling of binary matrices with fixed row sums and column sums for the fast Curveball algorithm

“…For larger matrices, the mixing time needs to be estimated. This can be performed by measuring the perturbation of each matrix in the Markov chain with respect to the initial matrix [17]. The mixing time is approximated by the step at which the perturbation score stabilizes.…”

“…O ur proof gives a theoretical justification for using the Curveball algorithm instead o f the more fam iliar sw itching method. Both sam ple uniform ly, but the C urveball algorithm is much faster [17].…”

“…This is in con tradiction with a comment made by Strona et al that removing all repeated states does not affect the Curveball algorithm. In the Supplementary Information of their paper [17], this argument is supported by presenting the transition matrix of a single example where all repeats are removed. However, it is a coincidence that sampling is uniform for this example.…”

“…Recently the Curveball algorithm was introduced as a much faster procedure to sample random binary matrices with fixed row sums and column sums [17]. Similar to swap methods, the Curveball algorithm randomizes a binary matrix by repeatedly making small changes.…”

Hide, Tallow and Terrapin: Gold Rush-Era Zooarchaeology at Thompson’s Cove (CA-SFR-186H), San Francisco, California

“…If recovery technique is biasing the assemblage then we hypothesize that the hand collected and 6.4 mm specimens should not be a nested subset of the 3.2 mm-recovered specimens. An open-source program NeD was used to quantify the NODF (nested overlap and decreasing fill) index and temperature (Strona et al 2014). Nestedness values range between 0, a perfectly nested fauna, and 100, a perfectly random fauna for temperature, but opposite for NODF (100 is a perfectly nested fauna) (Almeida-Neto et al 2008;Guimaraes and Guimaraes 2006).…”