Understanding Cloud Data Using Approximate String Matching and Edit Distance

Abstract

For health and human services, fraud detection and other security services, identity resolution is a core requirement for understanding big data in the cloud. Due to the lack of a globally unique identifier and captured typographic differences for the same identity, identity resolution has high spatial and temporal complexities. We propose a filter and verify method to substantially increase the speed of approximate string matching using edit distance. This method has been found to be almost 80 times faster (130 times when combined with other optimizations) than Damerau-Levenshtein edit distance and preserves all approximate matches. Our method creates compressed signatures for data fields and uses Boolean operations and an enhanced bit counter to quickly compare the distance between the fields. This method is intended to be applied to data records whose fields contain relatively shortlength strings, such as those found in most demographic data. Without loss of accuracy, the proposed Fast Bitwise Filter will provide substantial performance gain to approximate string comparison in database, record linkage and deduplication data processing systems.

MeDict: Health Dictionary Application Using Damerau-Levenshtein Distance Algorithm

“…Filters can make the system more efficient by removing unnecessary comparisons. One of the most common methods is length filtering, where the difference in the length of the two strings s and t must not be greater than k [7]. The algorithm of length filtering can be seen in Table 2.…”

“…Damerau-Levenshtein Distance algorithm is a development of the Levenshtein Distance algorithm. Damerau extended Levenshtein distance to also detect transposition errors and treat them as one edit operation [7]. Therefore Damerau-Levenshtein calculates the minimum insertion, deletion, substitution, and transposition operations to convert one word into another.…”

“…However, research by Sutisna and Adisantoso proved that spelling correction using the Damerau-Levenshtein Distance algorithm can improve a search engine performance by 22% rather than using the Levenshtein Distance algorithm [6]. Research by Jupin, Shi, and Obradovic proved that the Damerau-Levenshtein Distance algorithm has a smaller number of errors (false positive) than the Jaro-Winkler Distance algorithm [7]. Vogler explained that the choice of a string distance algorithm depends on the problem situation being encountered.…”

Damerau Levenshtein Distance for Indonesian Spelling Correction

“…The processes used on the Levenshtein distance algorithm are insertion, deletion, substitution. While, in the damerau Levenshtein distance algorithm, the operation used is almost the same as Levenshtein distance, but with the addition of the transposition operation between two characters [5]. Damerau Levenshtein does not distinguish between these four operations.…”

Identity Tracking in Big Data

“…An enhanced bit counting method [17] can be used to count the 1's in that only iterates once for each 1 in , such that = (1 ′ ). If is greater than 2 , the edit distance cannot be less than or equal to [18]. The complexity of the Bitwise Signature Filter is ( ) and for a dataset with strings it is ( ).…”