Double-array compression by pruning twin leaves and unifying common suffixes

Oono, Masaki; Morita, Ken; Sumitomo, Toru; Aoe, Jun-ichi

doi:10.1109/icoci.2006.5276476

Cited by 5 publications

(3 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Another approach to compress the double‐array is to employ compact trie forms. The minimal‐prefix (MP) trie 40,41 is an often‐used form, 4,7,16 and various methods to efficiently implement the MP‐trie using the double‐array have been proposed 17,42‐44 . However, the MP‐trie is specialized for dictionary lookups, which are done by traversing a trie from the root to a leaf.…”

Section: Related Workmentioning

confidence: 99%

“…The minimal-prefix (MP) trie 40,41 is an often-used form, 4,7,16 and various methods to efficiently implement the MP-trie using the double-array have been proposed. 17,[42][43][44] However, the MP-trie is specialized for dictionary lookups, which are done by traversing a trie from the root to a leaf. Other compact forms such as double-tries 45 and directed acyclic word graphs 20 cannot be applied to AC automata in principle.…”

Section: Application Example: Japanese Tokenizationmentioning

confidence: 99%

See 1 more Smart Citation

Engineering faster double‐array Aho–Corasick automata

Kanda¹,

Akabe²,

Oda

2023

Softw Pract Exp

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This article studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that comprehensive descriptions and experimental analyses on their ideas are not provided. Engineers face difficulties in implementing an efficient DAAC. In this article, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github. com/daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Application Example: Japanese Tokenizationmentioning

confidence: 99%

Engineering faster double‐array Aho–Corasick automata

Kanda¹,

Akabe²,

Oda

2023

Softw Pract Exp

View full text Add to dashboard Cite

show abstract

“…Another approach to compress the double-array is to employ compact trie forms. The minimal-prefix (MP) trie [3,13] is an often-used form [2,26,49], and various methods to efficiently implement the MP-trie using the doublearray have been proposed [12,24,48,50]. However, the MP-trie is specialized for dictionary lookups, which are done by traversing a trie from the root to a leaf.…”

Section: Double-arraymentioning

confidence: 99%

Engineering faster double-array Aho-Corasick automata

Kanda¹,

Akabe²,

Oda³

2022

Preprint

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This paper studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that their ideas are not aggregated. Since comprehensive descriptions and experimental analyses are unavailable, engineers face difficulties in implementing an efficient DAAC.In this paper, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github.com/ daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract