Detection of encrypted streams for egress monitoring

Malhotra, Paras

doi:10.31274/rtd-180813-15816

Cited by 9 publications

(19 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Dorfinger's Master thesis [20] proposes a system for discriminating encrypted and non-encrypted traffic, to ensure that all communications from a target network are encrypted. Similar approaches were also proposed by Mamun et al [31] and Malhotra [30]. Zhang et al proposed an entropy-based classifier for the identification of botnet traffic [42].…”

Section: Entropy-based Encryption Detectionmentioning

confidence: 82%

“…The χ 2 test is a simple statistical test to measure goodness of fit. It has been widely applied to distinguish compressed and encrypted content [12,30,34]. Given a set of samples, it measures how well the distribution of such samples follows a given distribution.…”

Section: χ 2 Testmentioning

confidence: 99%

“…In these settings, the metadata that is required by parsers is not present or is incomplete [35]. Given this issue, a number of works have been looking at alternative tests to distinguish between encrypted and compressed content [12,14,23,30,32,34,41]. While these works have the potential to be useful, there has been limited evaluation of their performance on a standardized dataset.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

EnCoD: Distinguishing Compressed and Encrypted File Fragments

Gaspari

Hitaj

Pagnotta

et al. 2020

Network and System Security

View full text Add to dashboard Cite

Reliable identification of encrypted file fragments is a requirement for several security applications, including ransomware detection, digital forensics, and traffic analysis. A popular approach consists of estimating high entropy as a proxy for randomness. However, many modern content types (e.g. office documents, media files, etc.) are highly compressed for storage and transmission efficiency. Compression algorithms also output high-entropy data, thus reducing the accuracy of entropy-based encryption detectors.Over the years, a variety of approaches have been proposed to distinguish encrypted file fragments from high-entropy compressed fragments. However, these approaches are typically only evaluated over a few, selected data types and fragment sizes, which makes a fair assessment of their practical applicability impossible. This paper aims to close this gap by comparing existing statistical tests on a large, standardized dataset. Our results show that current approaches cannot reliably tell apart encryption and compression, even for large fragment sizes. To address this issue, we design EnCoD, a learning-based classifier which can reliably distinguish compressed and encrypted data, starting with fragments as small as 512 bytes. We evaluate EnCoD against current approaches over a large dataset of different data types, showing that it outperforms current state-of-the-art for most considered fragment sizes and data types.

show abstract

Section: Entropy-based Encryption Detectionmentioning

confidence: 82%

Section: χ 2 Testmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

EnCoD: Distinguishing Compressed and Encrypted File Fragments

Gaspari

Hitaj

Pagnotta

et al. 2020

Network and System Security

View full text Add to dashboard Cite

show abstract

“…Encrypted data is known to be more uniformly distributed than unencrypted data [31,44]. Therefore, a number of approaches rely on this characteristic to locate cryptographic keys stored in memory and file system dumps [43].…”

Section: Related Workmentioning

confidence: 99%

“…al. [31], most traffic analysis tools analyse the contents of the data packets but they are not capable of determining whether data are encrypted. According to their research findings, Shannon entropy can be used to accurately classify low entropy streams.…”

Section: Related Workmentioning

confidence: 99%

HEDGE: Efficient Traffic Classification of Encrypted and Compressed Packets

Casino

Choo

Patsakis

2019

IEEE Trans.Inform.Forensic Secur.

View full text Add to dashboard Cite

As the size and source of network traffic increase, so does the challenge of monitoring and analysing network traffic. Therefore, sampling algorithms are often used to alleviate these scalability issues. However, the use of high entropy data streams, through the use of either encryption or compression, further compounds the challenge as current state of the art algorithms cannot accurately and efficiently differentiate between encrypted and compressed packets. In this work, we propose a novel traffic classification method named HEDGE (High Entropy DistinGuishEr) to distinguish between compressed and encrypted traffic. HEDGE is based on the evaluation of the randomness of the data streams and can be applied to individual packets without the need to have access to the entire stream. Findings from the evaluation show that our approach outperforms current state of the art. We also make available our statistically sound dataset, based on known benchmarks, to the wider research community.

show abstract

Reliable detection of compressed and encrypted data

Gaspari

Hitaj

Pagnotta

et al. 2022

Neural Comput & Applic

View full text Add to dashboard Cite

Several cybersecurity domains, such as ransomware detection, forensics and data analysis, require methods to reliably identify encrypted data fragments. Typically, current approaches employ statistics derived from byte-level distribution, such as entropy estimation, to identify encrypted fragments. However, modern content types use compression techniques which alter data distribution pushing it closer to the uniform distribution. The result is that current approaches exhibit unreliable encryption detection performance when compressed data appear in the dataset. Furthermore, proposed approaches are typically evaluated over few data types and fragment sizes, making it hard to assess their practical applicability. This paper compares existing statistical tests on a large, standardized dataset and shows that current approaches consistently fail to distinguish encrypted and compressed data on both small and large fragment sizes. We address these shortcomings and design EnCoD, a learning-based classifier which can reliably distinguish compressed and encrypted data. We evaluate EnCoD on a dataset of 16 different file types and fragment sizes ranging from 512B to 8KB. Our results highlight that EnCoD outperforms current approaches by a wide margin, with accuracy ranging from $$\sim 82\%$$ ∼ 82 % for 512B fragments up to $$\sim 92\%$$ ∼ 92 % for 8KB data fragments. Moreover, EnCoD can pinpoint the exact format of a given data fragment, rather than performing only binary classification like previous approaches.

show abstract

Detection of encrypted streams for egress monitoring

Cited by 9 publications

References 7 publications

EnCoD: Distinguishing Compressed and Encrypted File Fragments

EnCoD: Distinguishing Compressed and Encrypted File Fragments

HEDGE: Efficient Traffic Classification of Encrypted and Compressed Packets

Reliable detection of compressed and encrypted data

Contact Info

Product

Resources

About