Automated Dataset Generation System for Collaborative Research of Cyber Threat Analysis

Abstract

The objectives of cyberattacks are becoming sophisticated, and attackers are concealing their identity by masquerading as other attackers. Cyber threat intelligence (CTI) is gaining attention as a way to collect meaningful knowledge to better understand the intention of an attacker and eventually predict future attacks. A systemic threat analysis based on data acquired from actual cyber incidents is a useful approach to generating intelligence for such an objective. Developing an analysis technique requires a high volume and fine quality data. However, researchers can become discouraged by an inaccessibility to data because organizations rarely release their data to the research community. Owing to a data inaccessibility issue, academic research tends to be biased toward techniques that develope steps of the CTI process other than analysis and production.In this paper, we propose an automated dataset generation system called CTIMiner. The system collects threat data from publicly available security reports and malware repositories. The data are stored in a structured format. We released the source codes and dataset to the public, including approximately 640,000 records from 612 security reports published from January 2008 to June 2019. In addition, we present a statistical feature of the dataset and techniques that can be developed using it. Moreover, we demonstrate an application example of the dataset that analyzes the correlation and characteristics of an incident. We believe our dataset will promote collaborative research on threat analysis for the generation of CTI.

Cyber risk and cybersecurity: a systematic review of data availability

“…Due to the rapidly changing nature of cyber risks, it is often impossible to obtain all information on them. Kim and Kim (2019) proposed an automated dataset generation system called CTIMiner that collects threat data from publicly available security reports and malware repositories. They released a dataset to the public containing about 640,000 records from 612 security reports published between January 2008 and 2019.…”

Cyber Threat Intelligence in Risk Management

“…The authors define in [4] CTI as a means of gathering knowledge to understand what the attacker wants and predict future attacks. CTI is used to achieve appropriate awareness of conditions and cyber threats can be countered by including CTI in defense systems.…”

Collecting Vulnerable Source Code from Open-Source Repositories for Dataset Generation

“…The need for datasets and their generation are recurrent topics related to several research fields. Thus, there are published works in research areas as varied as radio signal processing [29], vehicular technology [30,31], vehicle-to-vehicle and vehicle-to-infrastructure wireless communication [32], computer vision [33] and pattern recognition [34], cyber threat intelligence [35], host intrusion detection [36], network intrusion detection system [37,38], smart grids [39], and software vulnerabilities [40][41][42][43][44][45], among many others.…”

“…Current approaches point in that direction. For instance, Daegeon et al's research [35] proposes a system for collecting threat data gathered from security reports and publicly available malware repositories.…”