A Framework for the Evaluation of Trainee Performance in Cyber Range Exercises

Andreolini, Mauro; Colacino, Vincenzo Giuseppe; Colajanni, Michele; Marchetti, Mirco

doi:10.1007/s11036-019-01442-0

Cited by 35 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our paper investigates the applicability of command-line data in educational assessment. Such data are collected in multiple state-of-the-art learning environments for cybersecurity training (Andreolini, Colacino, Colajanni, & Marchetti, 2019;Labuschagne & Grobler, 2017;Tian et al, 2018;Weiss, Turbak, Mache, & Locasto, 2017).…”

Section: Using Data For Student Assessment In Cybersecuritymentioning

confidence: 99%

Student assessment in cybersecurity training automated by pattern mining and clustering

et al. 2022

View full text Add to dashboard Cite

Hands-on cybersecurity training allows students and professionals to practice various tools and improve their technical skills. The training occurs in an interactive learning environment that enables completing sophisticated tasks in full-fledged operating systems, networks, and applications. During the training, the learning environment allows collecting data about trainees’ interactions with the environment, such as their usage of command-line tools. These data contain patterns indicative of trainees’ learning processes, and revealing them allows to assess the trainees and provide feedback to help them learn. However, automated analysis of these data is challenging. The training tasks feature complex problem-solving, and many different solution approaches are possible. Moreover, the trainees generate vast amounts of interaction data. This paper explores a dataset from 18 cybersecurity training sessions using data mining and machine learning techniques. We employed pattern mining and clustering to analyze 8834 commands collected from 113 trainees, revealing their typical behavior, mistakes, solution strategies, and difficult training stages. Pattern mining proved suitable in capturing timing information and tool usage frequency. Clustering underlined that many trainees often face the same issues, which can be addressed by targeted scaffolding. Our results show that data mining methods are suitable for analyzing cybersecurity training data. Educational researchers and practitioners can apply these methods in their contexts to assess trainees, support them, and improve the training design. Artifacts associated with this research are publicly available.

show abstract

Section: Using Data For Student Assessment In Cybersecuritymentioning

confidence: 99%

Student assessment in cybersecurity training automated by pattern mining and clustering

et al. 2022

View full text Add to dashboard Cite

show abstract

“…Still, it addresses a different application domain -puzzle-based cybersecurity training, aiming to utilize the specific data properties to deal with process mining challenges. Some papers also directly address the utilization of process mining for the analysis of hands-on cybersecurity exercises [1,30,31,35]. These approaches demonstrate that directed graphs carefully constructed from command history can provide useful information about trainees' behavior.…”

Section: Related Workmentioning

confidence: 99%

Process Mining Analysis of Puzzle-Based Cybersecurity Training

Macak,

Oslejsek,

Buhnova

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…The system monitored terminal input and output and compared them with task milestones to automatically assess student progress. Similarly [34], [35] also implemented host-based logging to reconstruct the students' progress. Although the goal of the above-mentioned works is similar to ours, their approach is complementary.…”

Section: Collection and Analysis Of Cybersecurity Education Datamentioning

confidence: 99%

Toolset for Collecting Shell Commands and Its Application in Hands-on Cybersecurity Training

Švábenský

Vykopal

Tovarňák

et al. 2021

2021 IEEE Frontiers in Education Conference (FIE)

View full text Add to dashboard Cite

This Full Paper in the Innovative Practice category presents and evaluates a technical innovation for handson classes. When learning cybersecurity, operating systems, or networking, students perform practical tasks using a broad range of command-line tools. Collecting and analyzing data about the command usage can reveal valuable insights into how students progress and where they make mistakes. However, few learning environments support recording and inspecting command-line inputs, and setting up an efficient infrastructure for this purpose is challenging. To aid engineering and computing educators, we share the design and implementation of an open-source toolset for logging commands that students execute on Linux machines. Compared to basic solutions, such as shell history files, the toolset's novelty and added value are threefold. First, its configuration is automated so that it can be easily used in classes on different topics. Second, it collects metadata about the command execution, such as a timestamp, hostname, and IP address. Third, all data are instantly forwarded to central storage in a unified, semi-structured format. This enables automated processing of the data, both in real-time and post hoc, to enhance the instructors' understanding of student actions. The toolset works independently of the teaching content, the training network's topology, or the number of students working in parallel. We demonstrated the toolset's value in two learning environments at four training sessions. Over two semesters, 50 students played educational cybersecurity games using a Linux command-line interface. Each training session lasted approximately two hours, during which we recorded 4439 shell commands. The semiautomated data analysis revealed different solution patterns, used tools, and misconceptions of students. Our insights from creating the toolset and applying it in teaching practice are relevant for instructors, researchers, and developers of learning environments. We provide the software and data resulting from this work so that others can use them in their hands-on classes.

show abstract

A Framework for the Evaluation of Trainee Performance in Cyber Range Exercises

Cited by 35 publications

References 16 publications

Student assessment in cybersecurity training automated by pattern mining and clustering

Student assessment in cybersecurity training automated by pattern mining and clustering

Process Mining Analysis of Puzzle-Based Cybersecurity Training

Toolset for Collecting Shell Commands and Its Application in Hands-on Cybersecurity Training

Contact Info

Product

Resources

About