Topology-based sparsification of graph annotations

Abstract

Motivation Since the amount of published biological sequencing data is growing exponentially, efficient methods for storing and indexing this data are more needed than ever to truly benefit from this invaluable resource for biomedical research. Labeled de Bruijn graphs are a frequently-used approach for representing large sets of sequencing data. While significant progress has been made to succinctly represent the graph itself, efficient methods for storing labels on such graphs are still rapidly evolving. Results In this article, we present RowDiff, a new technique for compacting graph labels by leveraging expected similarities in annotations of vertices adjacent in the graph. RowDiff can be constructed in linear time relative to the number of vertices and labels in the graph, and in space proportional to the graph size. In addition, construction can be efficiently parallelized and distributed, making the technique applicable to graphs with trillions of nodes. RowDiff can be viewed as an intermediary sparsification step of the original annotation matrix and can thus naturally be combined with existing generic schemes for compressed binary matrices. Experiments on 10 000 RNA-seq datasets show that RowDiff combined with multi-BRWT results in a 30% reduction in annotation footprint over Mantis-MST, the previously known most compact annotation representation. Experiments on the sparser Fungi subset of the RefSeq collection show that applying RowDiff sparsification reduces the size of individual annotation columns stored as compressed bit vectors by an average factor of 42. When combining RowDiff with a multi-BRWT representation, the resulting annotation is 26 times smaller than Mantis-MST. Availability and implementation RowDiff is implemented in C++ within the MetaGraph framework. The source code and the data used in the experiments are publicly available at https://github.com/ratschlab/row_diff.

Lossless indexing with counting de Bruijn graphs

“…Depending on the number of k-mers and files, this matrix can have up to ∼ 10 12 rows (corresponding to distinct k-mers) and ∼ 10 7 columns (corresponding to different files or, in general, labels) [22]. However, it can be highly compressed thanks to its sparsity [32,3,23,2,14].…”

“…Leveraging similarity of annotations of neighboring nodes For the case of binary annotations, transformations assuming likely similarity between annotations of adjacent nodes in the graph and replacing them with relative differences have been explored in Mantis-MST [2] and RowDiff [14]. The RowDiff algorithm conceptually consists of two parts.…”

“…Approaches for representing relations between k -mers and input files have been extensively explored in the past decade ( Iqbal et al 2012 ; Almodaresi et al 2017 , 2020 ; Muggli et al 2017 ; Karasikov et al 2020b ; Danciu et al 2021 ). Motivated by the experiment discovery problem, which is to find a sequencing library within a large collection based on a query pattern, these methods encode binary metadata attributes (e.g., the membership of a k -mer to a certain sequence or file) in a sparse binary matrix.…”

Lossless Indexing with Counting de Bruijn Graphs

“…Approaches for representing relations between k-mers and input files have been extensively explored in the past decade [20,32,3,23,2,14]. Motivated by the experiment discovery problem, which is to find a sequencing library within a large collection based on a query pattern, these methods encode binary metadata attributes (e.g., the membership of a k-mer to a certain sequence or file) in a sparse binary matrix.…”

“…Depending on the number of k-mers and files, this matrix can have up to ∼ 10 12 rows (corresponding to distinct k-mers) and ∼ 10 7 columns (corresponding to different files or, in general, labels) [22]. However, it can be highly compressed thanks to its sparsity [32,3,23,2,14].…”