Sub-file Hashing Strategies for Fast Contraband Detection

Abstract

Traditional digital forensics processes do not scale well with the huge quantities of data present in a modern investigation, resulting in large investigative backlogs for many law enforcement agencies. Data reduction techniques are required for fast and effective digital forensics triage, and to reduce the time taken to conduct an investigation. This work explores the potential of sub-file cryptographic hashing strategies, where small fragments of files are hashed in lieu of processing the file in its entirety, for contraband detection. Results show that subfile hashing techniques perform well, particularly on solid state media, while also retaining a high degree of discriminating power. Such strategies may offer an opportunity to take advantage of the performance characteristics of non-mechanical media, streamlining future investigations and greatly reducing investigation times.

Reducing the Impact of Network Bottlenecks on Remote Contraband Detection

“…As the sub-file hashing techniques in this paper have previously been shown to produce unique signatures for the full versions of the datasets [20], their discriminating power will not be discussed here. Instead, the focus will be on the performance merits of both approaches.…”

“…Generalised sub-file signatures, which are not file type specific, were explored in McKeown et al [20], with an evaluation of signature generation techniques which use both the beginning and end of a file. Reading from the beginning of the file failed to generate unique signatures for two large datasets, even when processing a large number of data blocks.…”

“…As the quantity of data on storage media continues to grow, reducing the amount of data to process will continue to be an important area of research. In this paper, we further explore the behavioural characteristics of the sub-file hashing strategies introduced in McKeown et al [20].…”

“…Prior work [20] demonstrates that reading as few as 4096 bytes of data from the end of a file can be unique at the million image scale, while reading a small number of bytes only from the start of the file is less effective. This sub-file approach was also shown to provide considerable speed gains over reading the full file, particularly as file sizes increase.…”

“…Timed benchmarks were conducted for 1-4 data blocks, corresponding to 4KiB, 8KiB, 12KiB and 16KiB, in order to assess the performance degradation caused by reading more data. While previous investigation has shown that 4KiB is sufficient to uniquely identify every unique image in the Flickr 1 Million dataset [20], larger block sizes were tested to demonstrate scalability in scenarios which require more data to generate unique signatures. As this acquisition takes places at the logical level, no slack space is included.…”