Constructing Flow Graphs from Procedural Cybersecurity Texts

Abstract

Following procedural texts written in natural languages is challenging. We must read the whole text to identify the relevant information or identify the instruction-flow to complete a task, which is prone to failures. If such texts are structured, we can readily visualize instruction-flows, reason or infer a particular step, or even build automated systems to help novice agents achieve a goal. However, this structure recovery task is a challenge because of such texts' diverse nature. This paper proposes to identify relevant information from such texts and generate information flows between sentences. We built a large annotated procedural text dataset (CTFW) in the cybersecurity domain (3154 documents). This dataset contains valuable instructions regarding software vulnerability analysis experiences. We performed extensive experiments on CTFW with our LM-GNN model variants in multiple settings. To show the generalizability of both this task and our method, we also experimented with procedural texts from two other domains (Maintenance Manual and Cooking), which are substantially different from cybersecurity. Our experiments show that Graph Convolution Network with BERT sentence embeddings outperforms BERT in all three domains.

CyberQ: Generating Questions and Answers for Cybersecurity Education Using Knowledge Graph-Augmented LLMs

“…The Thirty-Eighth AAAI Conference on Artificial Intelligence (AAAI-24) and novice learners. Structured flow graphs were constructed from Capture-The-Flag (CTF) procedural cybersecurity texts (Pal et al 2021) to teach students vulnerability analysis. Knowledge graphs were proposed to guide students to work on cybersecurity projects (Deng et al 2019;Deng, Zeng, and Huang 2021).…”

“…Knowledge graphs provide precise but incomplete and nongeneralizable knowledge (Abu-Salih 2021). There is a potential synergy between LLMs and KGs, with structured knowledge from KGs reducing LLM inaccuracies (Pan et al 2023;Agrawal et al 2023a). Dialog models like LaMDA (Thoppilan et al 2022) have used task-specific queries to access structured knowledge through fine-tuning.…”

The Role of Semantic Parsing in Understanding Procedural Text

“…Recent research has focused on procedural reasoning as tracking entities throughout a procedural text. Datasets such as Propara (Tandon et al, 2020), Recipes , Procedural Cyber-Security text (Pal et al, 2021), and OpenPI (Tandon et al, 2020) are in the same direction. Procedural reasoning can also be influential in addressing causal reasoning (WIQA) , story understanding (Trip) (Storks et al, 2021), and abstractive multi-modal question answering (RecipeQA) (Yagcioglu et al, 2018).…”

“…Procedural reasoning is the ability to track entities and understand their evolution given a sequence of actions (Tandon et al, 2020). This kind of reasoning is crucial in understanding recipes Yagcioglu et al, 2018), manuals and tutorials (Tandon et al, 2020;Wu et al, 2022), cybersecurity text (Pal et al, 2021), natural events (Tandon et al, 2020), and even stories (Storks et al, 2021). An example of a procedural text in the natural event domain, its entities of interest, and their state changes are shown in Figure 1.…”