A framework for space-efficient variable-order Markov models

Abstract

Motivation: Markov models with contexts of variable length are widely used in bioinformatics for representing sets of sequences with similar biological properties. When models contain many long contexts, existing implementations are either unable to handle genome-scale training datasets within typical memory budgets, or they are optimized for specific model variants and are thus inflexible. Results: We provide practical, versatile representations of variable-order Markov models and of interpolated Markov models, that support a large number of context-selection criteria, scoring functions, probability smoothing methods, and interpolations, and that take up to 4 times less space than previous implementations based on the su x array, regardless of the number and length of contexts, and up to 10 times less space than previous trie-based representations, or more, while matching the size of related, state-of-the-art data structures from Natural Language Processing.We describe how to further compress our indexes to a quantity related to the redundancy of the training data, saving up to 90% of their space on repetitive datasets, and making them become up to 60 times smaller than previous implementations ⇤ based on the su x array. Finally, we show how to exploit constraints on the length and frequency of contexts to further shrink our compressed indexes to half of their size or more, achieving data structures that are 100 times smaller than previous implementations based on the su x array, or more. This allows variable-order Markov models to be trained on bigger datasets and with longer contexts on the same hardware, thus possibly enabling new applications. Availability and implementation:https://github.com/jnalanko/VOMM

Algorithms and Complexity on Indexing Founder Graphs

“…The bidirectional index can then be used to construct both of the required tree topologies in O(n log σ ) time using bidirectional extension operations and the counter-based topology construction method of Belazzougui et al [38]. See the supplement of a paper on variable order Markov models by Cunial et al [43] for more details on the construction of the tree topologies. The topologies are then indexed for various navigational operations required by the contraction operation described in [39].…”

Fast and compact matching statistics analytics

“…Given an array MS S,T and a user-defined threshold τ , let a thresholded matching statis-tics array MS S,T,τ be such that MS S,T,τ [i] = MS S,T [i] if MS S,T [i] ≥ τ , and MS S,T,τ [i] equals an arbitrary (possibly negative) value smaller than τ otherwise 2 . This notion is symmetrical to the one defined by [9], which discards instead long MS values in order to prune the suffix tree topologies and to make the data structures smaller. Given an encoder δ, we are interested in the MS S,T,τ array whose ms S,T,τ bitvector takes the smallest amount of space when encoded with δ.…”

Algorithms and Complexity on Indexing Founder Graphs

“…The bidirectional index can then be used to construct both of the required tree topologies in O(n log σ) time using bidirectional extension operations and the counter-based topology construction method of Belazzougui et al [28]. See the supplement of a paper on variable order Markov models by Cunial et al [33] for more details on the construction of the tree topologies. The topologies are then indexed for various navigational operations required by the contraction operation described in [29].…”