Inferring Statistically Significant Hidden Markov Models

Yu, Lu; Schwier, Jason; Craven, Ryan; Brooks, Richard; Griffin, Christopher

doi:10.1109/tkde.2012.93

Cited by 24 publications

(20 citation statements)

References 26 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We used the approach from [33] to determine both the significance of the models and the volume of data necessary for creating a significant model. For the Italian data, a reconstruction with a string length L = 3 was possible.…”

Section: English and Italian Detectionmentioning

confidence: 99%

“…We followed the process described by the flowchart in Figure 2.11 shows the HMM constructed for the first 200,000 packets. The model confidence test [33] was then run on the model and showed that the required amount of data kept increasing with each set. This was due to the noise (circuit failures and packet drops, etc) of the Tor connection.…”

Section: Experiments Setupmentioning

confidence: 99%

“…To use the z-test in this manner, we propose a simple algorithm to perform on-line testing of the observation sequence [33]. The algorithm determines if a constructed model statistically represents a data stream in the process of being collected.…”

Section: Model Confidence Algorithmmentioning

confidence: 99%

“…In [33], we provide simple illustrations of the concept of model confidence algorithm. The results of these examples illustrate the point that if an insufficient amount of data samples are used, the algorithm only creates a model representative of the data, not of the underlying process.…”

Section: Model Confidence Algorithmmentioning

confidence: 99%

“…Inter-packet delays were collected and used to infer HMMs. We applied a model confidence test [33] to check if the inferred model is representative of the underlying process. By restricting the communication protocol model to the subset of the HMM that was statistically significant, we were able to infer a HMM that was a faithful representation of the underlying protocol.…”

mentioning

confidence: 99%

See 4 more Smart Citations

Detecting Hidden Communications Protocols

Brooks¹

2013

Self Cite

View full text Add to dashboard Cite

Standard Form 298 (Rev. 8/98) REPORT DOCUMENTATION PAGEPrescribed by ANSI Std. Z39.18 Form Approved OMB No. 0704-0188The public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing the burden, to the Department of Defense, Executive Services and Communications Directorate (0704-0188). Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ORGANIZATION. 1. REPORT DATE (DD-MM-YYYY)2 The work funded by the grant is structured in three parts: We analyzed the vulnerability of the current generation anonymity tools to traffic analysis attacks. We specifically concentrate on SSH security and The Onion Router (Tor) anonymity tools. Our analysis used deterministic hidden Markov models (HMMs). We used traffic timing data to analyze one of the most sophisticated and popular types of cybercrime tools -botnet. We presented two botnet detection methods: centralized botnet traffic detection using HMMs and probabilistic context-free grammars (PCFGs) for centralized and P2P botnet traffic detection. Finally, a hybrid network security scheme that combines the advantages of widely deployed security technologies (intrusion detection systems (IDS) and honeypots) was proposed. The scheduling problem of the security system was modeled as an average decentralized partially observable Markov decision process (DEC-POMDP) and solved using our nonlinear programming (NLP)-based solution method.

show abstract

Section: English and Italian Detectionmentioning

confidence: 99%

Section: Experiments Setupmentioning

confidence: 99%

Section: Model Confidence Algorithmmentioning

confidence: 99%

Section: Model Confidence Algorithmmentioning

confidence: 99%

mentioning

confidence: 99%

See 3 more Smart Citations

Detecting Hidden Communications Protocols

Brooks¹

2013

Self Cite

View full text Add to dashboard Cite

show abstract

Scrybe: A Second-Generation Blockchain Technology with Lightweight Mining for Secure Provenance and Related Applications

Worley

Brooks

et al. 2020

Advances in Information Security

View full text Add to dashboard Cite

Stochastic Tools for Network Intrusion Detection

Brooks

2018

Proceedings of International Symposium on Sensor Networks, Systems and Security

Self Cite

View full text Add to dashboard Cite

With the rapid development of Internet and the sharp increase of network crime, network security has become very important and received a lot of attention. We model security issues as stochastic systems. This allows us to find weaknesses in existing security systems and propose new solutions. Exploring the vulnerabilities of existing security tools can prevent cyber-attacks from taking advantages of the system weaknesses. We propose a hybrid network security scheme including intrusion detection systems (IDSs) and honeypots scattered throughout the network. This combines the advantages of two security technologies. A honeypot is an activity-based network security system, which could be the logical supplement of the passive detection policies used by IDSs. This integration forces us to balance security performance versus cost by scheduling device activities for the proposed system. By formulating the scheduling problem as a decentralized partially observable Markov decision process (DEC-POMDP), decisions are made in a distributed manner at each device without requiring centralized control. The partially observable Markov decision process (POMDP) is a useful choice for controlling stochastic systems. As a combination of two Markov models, POMDPs combine the strength of hidden Markov Model (HMM) (capturing dynamics that depend on unobserved states) and that of Markov decision process (MDP) (taking the decision aspect into account). Decision making under uncertainty is used in many parts of business and science. We use here for security tools. We adopt a high-quality approximation solution for finite-space POMDPs with the average cost criterion, and their extension to DEC-POMDPs. We show how this tool could be used to design a network security framework.

show abstract

Inferring Statistically Significant Hidden Markov Models

Cited by 24 publications

References 26 publications

Detecting Hidden Communications Protocols

Detecting Hidden Communications Protocols

Scrybe: A Second-Generation Blockchain Technology with Lightweight Mining for Secure Provenance and Related Applications

Stochastic Tools for Network Intrusion Detection

Contact Info

Product

Resources

About