Singularity: pattern fuzzing for worst case complexity

Abstract

We describe a new blackbox complexity testing technique for determining the worst-case asymptotic complexity of a given application. The key idea is to look for an input pattern Ðrather than a concrete inputÐ that maximizes the asymptotic resource usage of the target program. Because input patterns can be described concisely as programs in a restricted language, our method transforms the complexity testing problem to optimal program synthesis. In particular, we express these input patterns using a new model of computation called Recurrent Computation Graph (RCG) and solve the optimal synthesis problem by developing a genetic programming algorithm that operates on RCGs. We have implemented the proposed ideas in a tool called Singularity and evaluate it on a diverse set of benchmarks. Our evaluation shows that Singularity can effectively discover the worst-case complexity of various algorithms and that it is more scalable compared to existing state-of-the-art techniques. Furthermore, our experiments also corroborate that Singularity can discover previously unknown performance bugs and availability vulnerabilities in real-world applications such as Google Guava and JGraphT. CCS CONCEPTS • Software and its engineering → Software performance; Software testing and debugging; • Security and privacy → Denial-ofservice attacks;

CrFuzz: fuzzing multi-purpose programs through input validation

“…TortoiseFuzz [36] and Ankou [26] propose new coverage evaluation techinques for better seed scheduling. Some studies [21,29,37] have focused on detecting algorithmic complexity vulnerabilities based on new coverage metrics such as resource usage or execution path length.…”

NetBOA

“…Rampart [24] targets the opposite use-case of algorithm attacks: it protects applications from exhaustive CPU exhaustion DoS attacks. Instead of finding concrete inputs that exhaust programs, Singularity [30] targets at a higher abstraction level: analyzing given program code it tries to find input patterns that exhaust the program application. The work of [25] synthesizes program code of network functions in order to find challenging network traffic configurations.…”

“…This paper makes the case for a data-driven approach to measuring and evaluating the performance of a network in an adaptive, self-driving manner. Indeed, existing work on automated benchmarking and fuzzing systems either (i) targets general computing systems and hence may not lend itself readily to be adopted in a networked setting [19,22,24,26,30], (ii) aims at verifying logical properties in networked systems related to policy-compliance of configurations and implementations and ignore performance [3,21], or (iii) requires human assistance and software source code [14,25] to guide the performance evaluations and experiments, relying on hand-crafted, and often proprietary, benchmark tools, inputs, and system settings [5,18,20]. We argue that in the context of selfdriving networks the performance evaluation tool itself must also be self-driving, taking into account the specialties of networked systems and the environment these systems are typically used in.…”

“…A machine-driven approach on the other hand alleviates the network operator from manually testing through the vast parameter and state space, eliminates the human from the loop and guarantees unbiased results, and, critically, fosters reproducibility. Third, a networked system requires valid input traffic, imposing stringent constraints on the type and characteristics of the input that can be posed to such a system, which is in contrast with conventional performance fuzzing that may use arbitrary inputs [19,22,24,26,30]. Fourth, in the context of self-driving networks, where the network continuously measures and adapts itself, a means is required that can thoroughly evaluate an updated component to ensure its correct functioning.…”

Adversarial Network Algorithm Benchmarking

“…Previous work on algorithm complexity attacks has already shown methods for generating challenging, often called adversary, algorithms inputs [8,10,13,17,18]. With the help of these inputs the authors were able to improve algorithm performance and close security holes.…”