Cost-based variable-length-gram selection for string collections to support approximate queries efficiently

Abstract

Approximate queries on a collection of strings are important in many applications such as record linkage, spell checking, and Web search, where inconsistencies and errors exist in data as well as queries. Several existing algorithms use the concept of "grams," which are substrings of strings used as signatures for the strings to build index structures. A recently proposed technique, called VGRAM, improves the performance of these algorithms by using a carefully chosen dictionary of variable-length grams based on their frequencies in the string collection. Since an index structure using fixed-length grams can be viewed as a special case of VGRAM, a fundamental problem arises naturally: what is the relationship between the gram dictionary and the performance of queries? We study this problem in this paper. We propose a dynamic programming algorithm for computing a tight lower bound on the number of common grams shared by two similar strings in order to improve query performance. We analyze how a gram dictionary affects the index structure of the string collection and ultimately the performance of queries. We also propose an algorithm for automatically computing a dictionary of high-quality grams for a workload of queries. Our experiments on real data sets show the improvement on query performance achieved by these techniques. To our best knowledge, this study is the first cost-based quantitative approach to deciding good grams for approximate string queries.

VChunkJoin: An Efficient Algorithm for Edit Similarity Joins

“…[28] is a survey on approximate string matching methods. Recent progress in the literature that is related to similarity joins includes similarity joins with various similarity or distance functions [29], [19], [5], [4], [8], [11], [12], [30], [31], [22], [32], [33], [21], similarity selection [34], [3], and selectivity estimation [35], [36], [37], [38], [39], [40].…”

“…Filters based on mismatching q-grams are proposed to further speed up the query processing [8]. Variable-length grams are also proposed [11], [12], which can be easily integrated into other algorithms and help to achieve better performance.…”

“…• VGram-Join is an algorithm based on variablelength grams for answering edit similarity selection queries [11], [12]. The VGRAM approach is known for its small index sizes and fast speed compared with fixed-length grams.…”

“…In order to scale to large datasets, all existing algorithms adopt the filter-and-verify paradigm, which first eliminates non-promising string pairs with relatively inexpensive computation and then performs verification by the edit distance computation. By far the most prevalent filtering approach is those based on grams; they can be further categorized into methods using fixed-length q-grams [6], [8] and recent proposals using variable-length grams (VGRAM) [11], [12].…”

“…(Line 5 in Algorithm 1) • Improved verification algorithm that uses L 1 distance based filtering. An interesting alternative is proposed recently in [11], [12]. The idea is to use variable-length grams, and it aims at striking a balance between rare and frequent tokens.…”