Hash-based carving: Searching media for complete files and file fragments with sector hashing and hashdb

Abstract

a b s t r a c tHash-based carving is a technique for detecting the presence of specific "target files" on digital media by evaluating the hashes of individual data blocks, rather than the hashes of entire files. Unlike whole-file hashing, hash-based carving can identify files that are fragmented, files that are incomplete, or files that have been partially modified. Previous efforts at hash-based carving have looked for evidence of a single file or a few files. We attempt hash-based carving with a target file database of roughly a million files and discover an unexpectedly high false identification rate resulting from common data structures in Microsoft Office documents and multimedia files. We call such blocks "nonprobative blocks." We present the HASH-SETS algorithm that can determine the presence of files, and the HASH-RUNS algorithm that can reassemble files using a database of file block hashes. Both algorithms address the problem of non-probative blocks and provide results that can be used by analysts looking for target data on searched media. We demonstrate our technique using the bulk_extractor forensic tool, the hashdb hash database, and an algorithm implementation written in Python. Published by Elsevier Ltd on behalf of DFRWS. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

An intelligent approach for examining and detecting target data fragments in suspected large storage drives

“…In addition to random sampling, rather exploiting the hash values of the entire file, the block‐based hashing approach has been effectively used in literature for identifying the existence of known contents on digital media or forensic images. The hash‐based carving technique presented in Garfinkel and McCarrin detects the presence of blacklisted files by evaluating and matching the hash values of overlapping blocks from disk images. For illustration, the features of hash‐based carving method were made compatible with the bulk_extractor forensic tool and hashdb database using an integrator python script.…”

“…For detection the comparison between the sector and block hashes can be accomplished by utilizing any of the available cryptographic hash algorithms (verification methods), for example, Message Digest (MD5) and Secure Hash Algorithm (SHA1). Here, we followed computation of MD5 hash values because of its computational efficiency and vast adaptability in the forensics community . Since the sector is the smallest unit to represent data in the disk drive, the evaluation of small‐sized sector or block is advised because file blocks should efficiently map with drive sectors .…”

LogDrive: a proactive data collection and analysis framework for time-traveling forensic investigation in IaaS cloud environments

“…One method to tackle the problem is the combination of parallel processing without file system information and conventional standalone forensic tools based on the file system information. In this paper, we employ sector hashing with random sampling [24][25][26][27]42] to search for an evidence file in a large amount of write logs by using a MapReduce cluster. Moreover, we propose the restoration function of a previous virtual block device at an arbitrary point in the past so that investigators will be able to perform detailed analysis of the restored disk using conventional forensic tools based on the file system information.…”

Significant data region identification and analysis using k‐means in large storage drive forensics

“…Since entropy can be similar for two different data items, researchers have used hash values because of their reduced collision probabilities to match known files from large datasets. For example, authors in Reference have used MD5 message digests for hash‐based file carving approach. Nevertheless, in this work, entropy is used to estimate the amount of information contained in each randomly selected sector.…”