Assessing a Decision Support Tool for SOC Analysts

Abstract

In recent years, many tools have been developed to understand attacks that make use of visualization, but few examples aims to predict real-world consequences. We have developed a visualization tool that aims to improve decision support during attacks. Our tool visualizes propagation of risks from IDS and AV-alert data by relating sensor alerts to Business Process (BP) tasks and machine assets: an important capability gap present in many Security Operation Centres (SOCs) today. In this paper we present a user study in which we evaluate the tool's usability and ability to deliver situational awareness to the analyst. Ten analysts from seven SOCs performed carefully designed tasks related to understanding risks and prioritising recovery decisions. The study was conducted in laboratory conditions, with simulated attacks, and used a mixed-method approach to collect data from questionnaires, eyetracking and voice-recorded interviews. The findings suggest that providing analysts with situational awareness relating to business priorities can help them prioritise response strategies. Finally, we provide an in-depth discussion on the wider questions related to user studies in similar conditions as well as lessons learned from our user study and developing a visualization tool of this type.

Mission‐centric decision support in cybersecurity via Bayesian Privilege Attack Graph

“…The preliminary steps towards evaluating such systems were investigated very recently, for example, in the work of Rodriguez‐Bermejo et al, 70 who propose an evaluation methodology for mission‐centric cyber situational awareness capabilities covering correct technical implementation, core functionality, and user acceptance. In another example, Happa et al 71 conducted a user evaluation study of decision support tools for SOC analysts. A recent complex decision support system for cybersecurity incident handling proposed by Husák et al 2 also resorted to case study and user evaluation.…”

“…A testbed could be set up to create a benchmark for the evaluation of the system in a controlled environment. However, field trials and more interactions with potential users shall give more insights than formal evaluation or performance analysis 70‐72 . We plan to extend the number of modeled missions and circle of their stakeholders to collect feedback on the decision support system and its features.…”

“…However, field trials and more interactions with potential users shall give more insights than formal evaluation or performance analysis. [70][71][72] We plan to extend the number of modeled missions and circle of their stakeholders to collect feedback on the decision support system and its features. For example, it would be interesting to find a balance between the level of details in the mission decomposition model and the cost to maintain it.…”

Understanding decision making in security operations centres: building the case for cyber deception technology

“…Further exploring the tacit aspects of decision making occurring within contemporary SOCs, Happa et al (2021) evaluate the effectiveness of a decision support tool for SOC analysts. The authors conduct a user study to assess the tool’s ability to support analysts in decision-making, triage, and prioritization tasks.…”

Lightweight Impact Assessment and Projection of Lateral Movement and Malware Infection

“…Recently, recommender systems were employed for the cybersecurity needs to help address the cyber attacks at the right place at the right time [3], [7], [8]. However, the cybersecurity teams are often understaffed or overloaded with work to successfully deploy all the tools proposed in recent research, namely in situations when the tools require non-trivial amounts of high-quality data or other inputs that are hard to collect [6], [9].…”