Challenges and Opportunities in Collaborative Vulnerability Research Workflows

Abstract

Recently, cyber reasoning systems demonstrated near-human performance characteristics when they autonomously identified, proved, and mitigated vulnerabilities in software during a competitive event. New research seeks to augment human vulnerability research teams with cyber reasoning system teammates in collaborative work environments. However, the literature lacks a concrete understanding of vulnerability research workflows and practices, limiting designers’, engineers’, and researchers’ ability to successfully integrate these artificially intelligent entities into teams. This paper contributes a general workflow model of the vulnerability research process, and identifies specific collaboration challenges and opportunities anchored in this model. Contributions were derived from a qualitative field study of work habits, behaviors, and practices of human vulnerability research teams. These contributions will inform future work in the vulnerability research domain by establishing an empirically-driven workflow model that can be adapted to specific organizational and functional constraints placed on individual and teams.

A Task Analysis of Static Binary Reverse Engineering for Security

“…It is a cycle that repeats as new leads for investigation are identified or removed from scope. Instead of a 3-phase process with clear milestones and transitions [11], we observed a range of process states that are visited frequently and iteratively during a given RE task, with little indication of how much progress the analyst was making toward a larger goal. Moreover, there was a high degree of variation across participants and their process sequences, indicating that the RE process is less formal or organized than previous representations suggest [11].…”

“…Tilley et al identified 3 phases of data gathering, knowledge generation, and information exploration [10]. These studies generalize across types of RE tasks, while Mullins et al focused on defining these phases for a specific RE task [11]. They identified 3 phases in vulnerability discovery: reconnaissance, analysis, and patch & proof.…”

“…Popular methods for studying the RE process include interviews and think aloud protocols to elicit process knowledge and expertise from experienced individuals. Some studies have collected cognitive information to gain these expert perspectives [8,9,11,13], but many of these studies allow participants to choose an example and walk through what they did or would have done. Unfortunately, people have more difficulty recalling process steps that did not result in useful information; thus, retrospective reports are less likely to capture some of the pain points.…”