Conway-Bromage-Lyndon (CBL): an exact, dynamic representation of<i>k</i>-mer sets

Martayan, Igor; Cazaux, Bastien; Limasset, Antoine; Marchet, Camille

doi:10.1101/2024.01.29.577700

Cited by 4 publications

(8 citation statements)

References 55 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Experimental evaluation of indexing. We compared time and memory requirements for processing both positive and negative queries of FMSI to state-of-the-art programs for indexing individual k -mer sets, namely, BWA 1 [38], a state-of-the-art aligner based on the FM index; for processing queries, we used the fastmap command [36], run with parameter w = 999999 on the simplitigs computed by ProphAsm [11], SBWT 2 [1], an index based on the spectral Burrows-Wheeler transform; we used the default plain-matrix variant as it achieves the best query times in [1] and added all reverse complements to the index, and CBL 3 [45], a very recent method based on smallest cyclic rotations of k -mers. We have run FMSI on the masked superstrings computed by KmerCamel [61], specifically the global and local greedy algorithms (local is run with d = 1).…”

Section: Experimental Evaluationmentioning

confidence: 99%

“…We compared FMSI against state-of-the-art single 𝑘-mer set indexes such as BWA [38], SBWT [1], SSHash [52], and CBL [46] and evaluated them on E. coli pan-genome (obtained as a 𝑘-mer union over E. coli genomes from the Fig. 1: Time and memory efficiency of 𝑘-mer membership queries.…”

Section: Implementation and Experimental Evaluationmentioning

confidence: 99%

“…-CBL [46] (https://github.com/imartayan/CBL, commit b3a1158), a very recent method based on smallest cyclic rotations of 𝑘-mers. The index was computed on canonical 𝑘-mers, to handle reverse complements, that is, we ran cbl build -c. We note that CBL needs to be compiled for every value of 𝑘 separately.…”

Section: F Experimental Evaluation Of Set Operationsmentioning

confidence: 99%

“…Very recently, the Conway-Bromage-Lyndon (CBL) structure [45] builds on the work of Conway and Bromage [19] and combines smallest cyclic rotations of k -mers with sparse bit-vector encodings, to yield a dynamic and exact k -mer index supporting set operations, such as union, intersection, and difference. Finally, we note that set operations can also be carried out using some k -mer lists and counters, e.g., [29, 32]; however, these methods are unable to exploit structural properties of k -mer sets such as the SLP.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Towards Efficientk-Mer Set Operations via Function-Assigned Masked Superstrings

Sladký,

Veselý,

Břinda

2024

Preprint

View full text Add to dashboard Cite

The exponential growth of genome databases calls for novel space-efficient algorithms for data compression and search. State-of-the-art approaches often rely on 𝑘-merization for data tokenization, yet efficiently representing and querying 𝑘-mer sets remains a significant challenge in bioinformatics. Our recent work has introduced the concept of masked superstring for compactly representing 𝑘-mer sets, designed without reliance on common structural assumptions on 𝑘-mer data. However, despite their compactness, the practicality of masked superstrings for set operations and membership queries was previously unclear. Here, we propose the 𝑓-masked superstring framework, which additionally integrates demasking functions 𝑓, enabling efficient 𝑘-mer set operations through concatenation. When combined with the FMS-index, a new index for 𝑓-masked superstrings based on a simplified FM-index, we obtain a versatile, compact data structure for 𝑘-mer sets. We demonstrate its power through the FMSI program, which, when evaluated on bacterial pan-genomic data, achieves memory savings of a factor of 3 to 10 compared to state-of-the-art single 𝑘-mer-set indexing methods such as SBWT and CBL. Our work presents a theoretical framework with promising practical advantages such as space-efficiency, demonstrating the potential of 𝑓-masked superstrings in 𝑘-mer-based methods as a generic data type.

show abstract

Section: Experimental Evaluationmentioning

confidence: 99%

Section: Implementation and Experimental Evaluationmentioning

confidence: 99%

Section: F Experimental Evaluation Of Set Operationsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Towards Efficientk-Mer Set Operations via Function-Assigned Masked Superstrings

Sladký,

Veselý,

Břinda

2024

Preprint

View full text Add to dashboard Cite

show abstract

“…We demonstrate how to construct subsets of k -mers using these tags and how to simplify query operations to reduce computational delays. We implement a solution using a recent k -mer set data-structure that was tailored for set operations [11]. We demonstrate that our method is capable of handling thousands of datasets efficiently.…”

Section: Introductionmentioning

confidence: 99%

Constrained enumeration ofk-mers from a collection of references with metadata

Ingels,

Martayan,

Salson

et al. 2024

Preprint

Self Cite

View full text Add to dashboard Cite

While recent developments ink-mers indexing methods have opened up many new possibilities, they still have limitations in handling certain types of queries, such as identifyingk-mers present in one dataset but absent in another. In this article, we present a framework for efficiently enumerating allk-mers within a collection of references that satisfy constraints related to their metadata tags. Our method involves simplifying the query beforehand to reduce computation delays; the construction of the solution itself is carried out using CBL, a recent data structure specifically dedicated to the optimised computation of set operations onk-mer sets. We provide an implementation to our solution and we demonstrate its capabilities using real genomic data (microbial and RNA-seq), and show examples of use cases to identifyk-mers of biological interest.FundingThis work is funded by a grant from the French ANR: Full-RNA ANR-22-CE45-0007. Igor Martayan is supported by a doctoral grant from ENS Rennes.

show abstract

Indexing All Life’s Known Biological Sequences

Karasikov¹,

Mustafa²,

Danciu³

et al. 2020

Preprint

108

View full text Add to dashboard Cite

The amount of biological sequencing data available in public repositories is growing exponentially, forming an invaluable biomedical research resource. Yet, making all this sequencing data searchable and easily accessible to life science and data science researchers is an unsolved problem. We present MetaGraph, a versatile framework for the scalable analysis of extensive sequence repositories. MetaGraph efficiently indexes vast collections of sequences to enable fast search and comprehensive analysis. A wide range of underlying data structures offer different practically relevant trade-offs between the space taken by an index and its query performance. Achieving compression ratios of up to 1,000-fold over the already compressed raw input data, MetaGraph indexes can represent the content of large sequencing archives in the working memory of a single compute server. We demonstrate our framework's scalability by indexing over 1.4 million whole genome sequencing (WGS) records from NCBI's Sequence Read Archive, representing a total input of more than three petabases. MetaGraph provides a flexible methodological framework allowing for index construction to be scaled from consumer laptops to distribution onto a cloud compute cluster for processing terabases to petabases of input data. Notably, processing of data sets ranging from 1 TB of raw WGS reads to 20 TB of human RNA-sequencing data results in indexes whose memory footprints are small enough to host on standard desktop workstations. Besides demonstrating the utility of MetaGraph indexes on key applications, such as experiment discovery, sequence alignment, error correction, and differential assembly, we make a wide range of indexes available as a community resource, including indexes of over 450,000 microbial WGS records, more than 110,000 fungi WGS records, and more than 40,000 whole metagenome sequencing records. A subset of these indexes is made available online for interactive queries. All indexes will be available for download and in the cloud. In total, indexes comprising more than 1 million sequencing records are available for download. As an example of our indexes' integrative analysis capabilities, we introduce the concept of differential assembly, which allows for the extraction of sequences present in a foreground set of samples but absent in a given background set. We apply this technique to differentially assemble contigs to identify pathogenic agents transfected via human kidney transplants. In a second example, we indexed more than 20,000 human RNA-Seq records from the TCGA and GTEx cohorts and use them to extract transcriptome features that are hard to characterize using a classical linear reference. We discovered over 200 trans-splicing events in GTEx and found broad evidence for tissue-specific non-A-to-I RNA-editing in GTEx and TCGA.

show abstract

Conway-Bromage-Lyndon (CBL): an exact, dynamic representation ofk-mer sets

Cited by 4 publications

References 55 publications

Towards Efficientk-Mer Set Operations via Function-Assigned Masked Superstrings

Towards Efficientk-Mer Set Operations via Function-Assigned Masked Superstrings

Constrained enumeration ofk-mers from a collection of references with metadata

Indexing All Life’s Known Biological Sequences

Contact Info

Product

Resources

About