An Efficient Similarity Digests Database Lookup – A Logarithmic Divide & Conquer Approach

Abstract

Investigating seized devices within digital forensics represents a challenging task due to the increasing amount of data. Common procedures utilize automated file identification, which reduces the amount of data an investigator has to examine manually. In the past years the research field of approximate matching arises to detect similar data. However, if n denotes the number of similarity digests in a database, then the lookup for a single similarity digest is of complexity of O(n). This paper presents a concept to extend existing approximate matching algorithms, which reduces the lookup complexity from O(n) to O(log(n)). Our proposed approach is based on the well-known divide and conquer paradigm and builds a Bloom filter-based tree data structure in order to enable an efficient lookup of similarity digests. Further, it is demonstrated that the presented technique is highly scalable operating a trade-off between storage requirements and computational efficiency. We perform a theoretical assessment based on recently published results and reasonable magnitudes of input data, and show that the complexity reduction achieved by the proposed technique yields a 2 20 -fold acceleration of look-up costs.

Similarity Digest Search: A Survey and Comparative Analysis of Strategies to Perform Known File Filtering Using Approximate Matching

“…As a form to mitigate the limitation of MRSH-NET [31], which only answers whether an object is present in the filter or not but does not identify the object, Breitinger et al [32] propose a new similarity digest search strategy based on the wellknown divide and conquer paradigm. In this approach, the authors build a Bloom filter-based tree data structure to store digests and efficiently locate similar objects.…”

“…However, MRSH-NET, BF-based tree, and MRSH-CF approaches do, according to the explanation presented below. According to Breitinger et al [32], the false positive probability for an object is calculated by = , where is the false positive probability for a single feature and the number of following features required to be found in the filter to be considered a match. While can be adjusted according to the desired false positive rate, is defined by…”

“…, where is the number of independent hash functions, the number of features inserted into the Bloom filter, and the filter size [32]. This way, the strategy decides when a match is found by a number of features present in the filter, and based on it, it can create more or less false positives depending on the parameters used by the strategy.…”

“…This way, when calculating the filter size and getting a result of 2 31.27 bits, for instance, we need to adjust their size for 2 32 . Although this modification can almost double the theoretical size of the filter in some cases, it is necessary for practical implementations.…”

A comparative analysis about similarity search strategies for digital forensics investigations

“…To mitigate the limitation of MRSH-NET in answering only membership queries, Breitinger, F. et al [18] proposed a new approach for performing similarity search. This strategy is based on the well-known divide and conquer paradigm.…”