Data-Driven Mathematical Approach for Removing Rare Features in Zero-Inflated Datasets

Abstract

Sparse feature tables, in which many features are present in very few samples, are common in big biological data (e.g., metagenomics, transcriptomics). Ignoring the problem of zero-inflation can result in biased statistical estimates and decrease power in downstream analyses. Zeros are also a particular issue for compositional data analysis using log-ratios since the log of zero is undefined. Researchers typically deal with zero-inflated data by removing low frequency features, but the thresholds for removal differ markedly between studies with little or no justification. Here, we present CurvCut, a data-driven mathematical approach to zero-inflated feature removal based on curvature analysis of a 'ball rolling down a hill', where the hill is a histogram of feature distribution. These histograms typically contain a point of regime change, a discontinuity with a sharp change in the characteristics of the distribution, that can be used as a cutoff point for low frequency feature removal that considers the data-specific nature of the feature distribution. Our results show that CurvCut works well across a variety of biological data types, including ones with both right- and left-skewed feature distributions, and rapidly generates clear visual results allowing researchers to select data-appropriate cutoffs for feature removal.

Genetic hypogonadal (Gnrh1hpg) mouse model uncovers influence of reproductive axis on maturation of the gut microbiome during puberty

“…Sequences classified as mitochondria were removed with qiime taxa filter-table, leaving 741 SVs in the feature table. To deal with sparse features, SVs not present in at least five samples were removed using CurvCut [34], resulting in a final count of 333 SVs.…”

“…Salmon was used to generate a count table of coding sequences (genes) for each sample [47]. Gene features not present in at least four samples were removed using CurvCut [34].…”

Genetic hypogonadal mouse model reveals niche-specific influence of reproductive axis and sex on intestinal microbial communities

“…Removal of likely contaminants resulted in 741 SVs in the fecal samples, 606 SVs in the cecum samples, 832 SVs in ileal samples, and 1554 SVs in duodenum samples. These SVs were then zero-filtered to remove very low-abundance SVs using the CurvCut heuristic approach [86], which suggested feature removal of SVs present in 3 or fewer samples, resulting in a final count of 333 fecal SVs, 309 cecum SVs, 286 ileal SVs, and 281 duodenum SVs.…”