A new compression method of double array for compact dictionaries

Fuketa, Masao; Morita, Ken; Sumitomo, Toru; Kashiji, Shinkaku; Atlam, El‐Sayed; Aoe, Jun-ichi

doi:10.1080/00207160410001714600

Cited by 5 publications

(5 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Some studies have represented the integers of BASE and CHECK in a compressed space other than Compact (described in Section 3.4). Fuketa et al 14 proposed partitioning one double‐array structure into several smaller ones to implement BASE and CHECK consisting of 2‐byte integers; however, as this approach produces many structures for a large trie, many pointers must be maintained to connect the structures. Fuketa et al 39 proposed eliminating BASE by introducing additional character mappings, although its applications are limited to fixed‐length keywords such as zip codes.…”

Section: Related Workmentioning

confidence: 99%

“…13 To achieve a higher performance in DAACs, it is essential to carefully design the data structure with regard to the target applications and data characteristics. In the three decades since the original idea of the double-array was proposed by Aoe, 4 many implementation techniques have been developed from different points of view, such as scalability, 7,14,15 cache efficiency, 16,17 construction speed, [18][19][20] and specializations. 10,11,20 In addition to these academic studies, many open-source libraries have been developed, for example, References 5,6,21, and 22, which sometimes contain original techniques.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Engineering faster double‐array Aho–Corasick automata

Kanda¹,

Akabe²,

Oda

2023

Softw Pract Exp

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This article studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that comprehensive descriptions and experimental analyses on their ideas are not provided. Engineers face difficulties in implementing an efficient DAAC. In this article, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github. com/daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Engineering faster double‐array Aho–Corasick automata

Kanda¹,

Akabe²,

Oda

2023

Softw Pract Exp

View full text Add to dashboard Cite

show abstract

“…Some studies have represented the integers of BASE and CHECK in a compressed space other than Compact (described in Section 3.4). Fuketa et al [18] proposed partitioning one double-array structure into several smaller ones to implement BASE and CHECK consisting of 2-byte integers; however, as this approach produces many structures for a large trie, many pointers must be maintained to connect the structures. Fuketa et al [17] proposed eliminating BASE by introducing additional character mappings, although its applications are limited to fixed-length keywords such as zip codes.…”

Section: Double-arraymentioning

confidence: 99%

“…To achieve a higher performance in DAACs, it is essential to carefully design the data structure with regard to the target applications and data characteristics. In the three decades since the original idea of the double-array was proposed by Aoe [2], many implementation techniques have been developed from different points of view, such as scalability [18,25,26], cache efficiency [24,49], construction speed [32,37,47], and specializations [36,44,47]. In addition to these academic studies, many open-source libraries have been developed, e.g., [20,28,40,46], which sometimes contain original techniques.…”

Section: Introductionmentioning

confidence: 99%

Engineering faster double-array Aho-Corasick automata

Kanda¹,

Akabe²,

Oda³

2022

Preprint

View full text Add to dashboard Cite

Multiple pattern matching in strings is a fundamental problem in text processing applications such as regular expressions or tokenization. This paper studies efficient implementations of double-array Aho-Corasick automata (DAACs), data structures for quickly performing the multiple pattern matching. The practical performance of DAACs is improved by carefully designing the data structure, and many implementation techniques have been proposed thus far. A problem in DAACs is that their ideas are not aggregated. Since comprehensive descriptions and experimental analyses are unavailable, engineers face difficulties in implementing an efficient DAAC.In this paper, we review implementation techniques for DAACs and provide a comprehensive description of them. We also propose several new techniques for further improvement. We conduct exhaustive experiments through real-world datasets and reveal the best combination of techniques to achieve a higher performance in DAACs. The best combination is different from those used in the most popular libraries of DAACs, which demonstrates that their performance can be further enhanced. On the basis of our experimental analysis, we developed a new Rust library for fast multiple pattern matching using DAACs, named Daachorse, as open-source software at https://github.com/ daac-tools/daachorse. Experiments demonstrate that Daachorse outperforms other AC-automaton implementations, indicating its suitability as a fast alternative for multiple pattern matching in many applications.

show abstract

“…An edge of the trie is represented by a character in previous researches related to the double array. As for the compression of the double array, there are methods dividing the trie [8], [9], a method removing BASE array [10], but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the double array method which represents an edge by n-byte.…”

Section: Introductionmentioning

confidence: 99%

A Retrieval Method for Double Array Structures by Using Byte N-Gram

Fuketa¹,

Morita²,

Aoe³

2014

IJTEF

Self Cite

View full text Add to dashboard Cite

Abstract-Retrieving keywords requires speed and compactness. A trie is one of the data structure to retrieve keywords, and the double array is one of the implementation methods for the trie. The retrieval algorithm for the double array is fast, and its data structure has compactness. An edge of the trie is represented by a character in previous researches related to the double array, but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the data structure and the retrieval algorithm for the double array which represents an edge by n-byte. This paper also proposes a method to compress CODE array. From the experimental results comparing with the original double array by using single-byte and multi-byte character sets, the size and the retrieval speed of the proposed method became 62-64% and 1.18-1.3 times, respectively. When the CODE is compressed, the sizes of the proposed method became 41-59%.Index Terms-Compression, double array, n-gram, trie. I. INTRODUCTIONIn ubiquitous environments such as smart phones and PDAs, the storage capacity is often limited. Retrieving keywords used in many applications requires speed and compactness. A trie is one of data structures to retrieve keywords. In the trie, common prefixes of stored keys are merged and each edge is labeled with a character consisting of keys. Because the trie can retrieve common prefix keywords and predictive keywords, it is used in information retrieval systems [1], natural language processing [2], IP address routing tables [3], and packet filtering [4]. Moreover, the trie is often used as an associative array [5] like a map class in C++ in order to improve the array by hash tables.A double array is one of the retrieval methods by using the trie. This method uses two arrays called BASE and CHECK, and it has speed and compactness [6], [7]. An edge of the trie is represented by a character in previous researches related to the double array. As for the compression of the double array, there are methods dividing the trie [8], [9], a method removing BASE array [10], but there are no researches discussing if the edge is represented n-gram. Therefore, this paper proposes the double array method which represents an edge by n-byte. This paper also proposes a method to compress CODE array by using the double array, because CODE array becomes big with n"s increasing.Section II describes the trie and the double array. Section Manuscript received September 23, 2013; revised November 25, 2013. This work was supported by JSPS KAKENHI Grant Number 24500118.The authors are with the Department of Information Science and Intelligent, University of Tokushima, Tokushima, Japan (e-mail: fuketa@is.tokushima-u.ac.jp, kam@is.tokushima-u.ac.jp, aoe@is.tokushima-u.ac.jp).III describes the proposed data structures and retrieval algorithms. Experimental evaluations are given in Section IV. Finally, Section V concludes the proposed algorithm and describes further works. II. DOUBLE ARRAYA trie is a tree structure to store some keys. T...

show abstract

A new compression method of double array for compact dictionaries

Cited by 5 publications

References 4 publications

Engineering faster double‐array Aho–Corasick automata

Engineering faster double‐array Aho–Corasick automata

Engineering faster double-array Aho-Corasick automata

A Retrieval Method for Double Array Structures by Using Byte N-Gram

Contact Info

Product

Resources

About