A Linear-Time Burrows-Wheeler Transform Using Induced Sorting

Okanohara, Daisuke; Sadakane, Kunihiko

doi:10.1007/978-3-642-03784-9_9

Cited by 64 publications

(42 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…While the inverted index can be built in almost the same space of the final index, our (F)WCSA prototype needs to build the suffix array first. Recent advances in building self-indexes within compressed space [González and Navarro 2008;Okanohara and Sadakane 2009;Hon et al 2009] should easily carry on to our scenario as well.…”

Section: Discussionmentioning

confidence: 99%

Word-based self-indexes for natural language text

Fariña

Brisaboa

Navarro

et al. 2012

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

The inverted index supports efficient full-text searches on natural language text collections. It requires some extra space over the compressed text that can be traded for search speed. It is usually fast for single-word searches, yet phrase searches require more expensive intersections. In this article we introduce a different kind of index. It replaces the text using essentially the same space required by the compressed text alone (compression ratio around 35%). Within this space it supports not only decompression of arbitrary passages, but efficient word and phrase searches. Searches are orders of magnitude faster than those over inverted indexes when looking for phrases, and still faster on single-word searches when little space is available. Our new indexes are particularly fast at counting the occurrences of words or phrases. This is useful for computing relevance of words or phrases.We adapt self-indexes that succeeded in indexing arbitrary strings within compressed space to deal with large alphabets. Natural language texts are then regarded as sequences of words, not characters, to achieve word-based self-indexes. We design an architecture that separates the searchable sequence from its presentation aspects. This permits applying case folding, stemming, removing stopwords, etc. as is usual on inverted indexes. Additional Key Words and Phrases: Self-indexes, compressed data structures, inverted indexes A preliminary partial version on this work appeared in Proc. SPIRE'08 [Brisaboa et al. 2008].

show abstract

Section: Discussionmentioning

confidence: 99%

Word-based self-indexes for natural language text

Fariña

Brisaboa

Navarro

et al. 2012

ACM Trans. Inf. Syst.

View full text Add to dashboard Cite

show abstract

“…This is the first time o(n lg n) time is obtained within compressed space. Other space-conscious results that achieve better time complexity (but more space) are Okanohara and Sadakane [32], who achieved optimal O(n) time within O(n lg σ lg lg σ n) bits, and Hon et al [24], who achieved O(n lg lg σ) time and O(n lg σ) bits.…”

Section: Burrows-wheeler Transformmentioning

confidence: 99%

Optimal Dynamic Sequence Representations

Navarro¹,

Nekrich²

2014

SIAM J. Comput.

View full text Add to dashboard Cite

We describe a data structure that supports access, rank and select queries, as well as symbol insertions and deletions, on a string S[1, n] over alphabet [1..σ] in time O(lg n/ lg lg n), which is optimal. The time is worst-case for the queries and amortized for the updates. This complexity is better than the best previous ones by a Θ(1 + lg σ/ lg lg n) factor. Our structure uses nH0(S) + O(n + σ(lg σ + lg 1+ε n)) bits, where H0(S) is the zero-order entropy of S and 0 < ε < 1 is any constant. This space redundancy over nH0(S) is also better, almost always, than that of the best previous dynamic structures, o(n lg σ)+O(σ(lg σ+lg n)). We can also handle general alphabets in optimal time, which has been an open problem in dynamic sequence representations.

show abstract

“…Further enhancements to this compression technique is possible using recent improvements in the computation of BWT by Okanohara and Sadakane [14], notably an approach with linear time complexity [17].…”

Section: Burrows-wheeler Transformationmentioning

confidence: 99%

Scalability of communicators and groups in MPI

Kamal

Mirtaheri

Wagner

2010

Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing

View full text Add to dashboard Cite

As the number of cores inside compute clusters continues to grow, the scalability of MPI (Message Passing Interface) is important to ensure that programs can continue to execute on an ever-increasing number of cores. One important scalability issue for MPI is the current implementation of communicators and groups. Communicators and groups are an integral part of MPI and play an essential role in the design and use of libraries. It is challenging to create an MPI implementation to support communicators and groups to scale to the hundreds of thousands of processes that are possible in today's clusters. In this paper we present the design and evaluation of techniques to support the scalability of communicators and groups in MPI.We have designed and implemented a fine-grain version of MPI, called FG-MPI based on MPICH2, that allows thousands of full-fledged MPI processes inside an operating system process. Using FG-MPI we can create hundreds and thousands of MPI processes, which allowed us to implement and evaluate solutions to the scalability issues associated with communicators. We describe techniques to allow for sharing of group information inside processes, which required a re-definition of the context-id and the design of scalable operations to create the communicators. A set plus permutation framework is introduced for storing group information that makes use of a variety of different representations. We also propose a set, instead of map, representation for MPI group objects that takes advantage of our framework. Performance results are given for the execution of a MPI benchmark program with upwards of 100,000 processes with communicators created for various groups of different sizes and types.

show abstract

A Linear-Time Burrows-Wheeler Transform Using Induced Sorting

Cited by 64 publications

References 16 publications

Word-based self-indexes for natural language text

Word-based self-indexes for natural language text

Optimal Dynamic Sequence Representations

Scalability of communicators and groups in MPI

Contact Info

Product

Resources

About