The Tularosa Study: An Experimental Design and Implementation to Quantify the Effectiveness of Cyber Deception

Abstract

The Tularosa study was designed to understand how defensive deception-including both cyber and psychological-affects cyber attackers. Over 130 red teamers participated in a network penetration task over two days in which we controlled both the presence of and explicit mention of deceptive defensive techniques. To our knowledge, this represents the largest study of its kind ever conducted on a professional red team population. The design was conducted with a battery of questionnaires (e.g., experience, personality, etc.) and cognitive tasks (e.g., fluid intelligence, working memory, etc.), allowing for the characterization of a "typical" red teamer, as well as physiological measures (e.g., galvanic skin response, heart rate, etc.) to be correlated with the cyber events. This paper focuses on the design, implementation, data, population characteristics, and begins to examine preliminary results.

Are Cyber Attackers Thinking Fast and Slow? Exploratory Analysis Reveals Evidence of Decision-Making Biases in Red Teamers

“…With enough time spent interacting with such a system, an attacker may eventually care less about correctly identifying machines. Results from a similar study employing deception, which the authors also have access to (Ferguson-Walter et al, 2019), an attacker noted: Such an effect greatly improves the chances that a defensive team will detect and mitigate an attack. Biases in attacker decision-making can lead to psychologically motivated improvements in defense.…”

“…No doubt all types of rigorous assessment of human performance in cyber operators remains important to conduct. We noted throughout that our assessment of biases here was exploratory; however, a new experiment (Ferguson-Walter et al, 2019) involving 138 professional red teamers will have biases assessed using an inter-rater system combining cyber and psychology SMEs.…”

Design Thinking for Cyber Deception

“…Creating decoy systems with large numbers of false assets as opposed to real ones can reduce the ability of an attacker to successfully target a real asset through reducing their chances, distracting them from real assets, and forcing them to switch attention and perform additional tasks which, in turn, slows them down [4] [28]. Further benefits of decoy systems can be the improved detection of attackers from their engagement with false assets and increasing their level of confusion about the network's credibility [4].…”

“…Further benefits of decoy systems can be the improved detection of attackers from their engagement with false assets and increasing their level of confusion about the network's credibility [4]. Decoy networks have been extensively tested through red-teaming experiments, including detailed exploration of red teamers' behaviour, personality and cognition, and physiological responses to cyber deception [28]. Use of host-based deception and deceptive messages has been shown in experiments to disrupt attacker decision-making, increase the time to conduct the deception and increase confusion within the attacker or attacking team [15].…”

“…In taking this approach we build on the work of Reid and Black [30] and their qualitative, inductive study. This research includes a discussion about the connection between technology, and the real-world context (referencing the work on warrants and digital footprints by [31], [32]) and highlights the need for dynamic cyber deception systems and methods [17], [22] and for deception tools that are both 'generic and expressive' [28].…”