Practical rearrangement methods for dynamic double‐array dictionaries

Kanda, Shigenobu; Fujita, Yukihisa; Morita, Ken; Fuketa, Masao

doi:10.1002/spe.2516

Cited by 5 publications

(10 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…, 𝑞 𝑀 , where 0 < 𝑞 1 < 𝑞 2 < • • • < 𝑞 𝑀 < 𝑁. Given a state having 𝑘 outgoing transitions with labels This modification limits the maximum state id to 2 24 − 1. To represent state ids up to 2 29 − 1 with the same space usage, we can employ a technique used in the Darts-clone library [46] that utilizes two types of offset values to represent exact and rough positions.…”

Section: Chaining Vacant Idsmentioning

confidence: 99%

“…Skipping search blocks Before introducing SkipForward and SkipDense, we present a technique to partition array elements into fixed-size blocks [24,46,52]. The partitioning modifies Equation (2) into…”

Section: Chaining Vacant Idsmentioning

confidence: 99%

“…Another approach to compress the double-array is to employ compact trie forms. The minimal-prefix (MP) trie [3,13] is an often-used form [2,26,49], and various methods to efficiently implement the MP-trie using the doublearray have been proposed [12,24,48,50]. However, the MP-trie is specialized for dictionary lookups, which are done by traversing a trie from the root to a leaf.…”

Section: Double-arraymentioning

confidence: 99%

“…To achieve a higher performance in DAACs, it is essential to carefully design the data structure with regard to the target applications and data characteristics. In the three decades since the original idea of the double-array was proposed by Aoe [2], many implementation techniques have been developed from different points of view, such as scalability [18,25,26], cache efficiency [24,49], construction speed [32,37,47], and specializations [36,44,47]. In addition to these academic studies, many open-source libraries have been developed, e.g., [20,28,40,46], which sometimes contain original techniques.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Engineering faster double-array Aho-Corasick automata

Kanda¹,

Akabe²,

Oda³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This paper studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that their ideas are not aggregated. Since comprehensive descriptions and experimental analyses are unavailable, engineers face difficulties in implementing an efficient DAAC.In this paper, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github.com/ daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract

Section: Chaining Vacant Idsmentioning

confidence: 99%

Section: Chaining Vacant Idsmentioning

confidence: 99%

Section: Double-arraymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Engineering faster double-array Aho-Corasick automata

Kanda¹,

Akabe²,

Oda³

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Its space usage is 2n log n bits in the best case, while supporting getchild in O(1) time; however, a double array for a large alphabet tends to be empirically sparse. Actually, we are only aware of dynamic double-array implementations handling byte characters (e.g., [37,61]). Judy [10] and ART (adaptive radix tree) [42] are trie representations that dynamically choose suitable data structures for the trie topology; however, both are also designed for byte characters.…”

Section: Representation Schemementioning

confidence: 99%

Dynamic Path-Decomposed Tries

Kanda

Köppl

Tabei

et al. 2019

Preprint

Self Cite

View full text Add to dashboard Cite

A keyword dictionary is an associative array whose keys are strings. Recent applications handling massive keyword dictionaries in main memory have a need for a space-efficient implementation. When limited to static applications, there are a number of highly-compressed keyword dictionaries based on the advancements of practical succinct data structures. However, as most succinct data structures are only efficient in the static case, it is still difficult to implement a keyword dictionary that is space efficient and dynamic. In this article, we propose such a keyword dictionary. Our main idea is to embrace the path decomposition technique, which was proposed for constructing cache-friendly tries. To store the path-decomposed trie in small memory, we design data structures based on recent compact hash trie representations. Exhaustive experiments on real-world datasets reveal that our dynamic keyword dictionary needs up to 68% less space than the existing smallest ones.

show abstract

Engineering faster double‐array Aho–Corasick automata

Kanda¹,

Akabe²,

Oda

2023

Softw Pract Exp

Self Cite

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This article studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that comprehensive descriptions and experimental analyses on their ideas are not provided. Engineers face difficulties in implementing an efficient DAAC. In this article, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github. com/daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract

Practical rearrangement methods for dynamic double‐array dictionaries

Cited by 5 publications

References 30 publications

Engineering faster double-array Aho-Corasick automata

Engineering faster double-array Aho-Corasick automata

Dynamic Path-Decomposed Tries

Engineering faster double‐array Aho–Corasick automata

Contact Info

Product

Resources

About