Finding code that explodes under symbolic evaluation

Abstract

Solver-aided tools rely on symbolic evaluation to reduce programming tasks, such as verification and synthesis, to satisfiability queries. Many reusable symbolic evaluation engines are now available as part of solver-aided languages and frameworks, which have made it possible for a broad population of programmers to create and apply solver-aided tools to new domains. But to achieve results for real-world problems, programmers still need to write code that makes effective use of the underlying engine, and understand where their code needs careful design to elicit the best performance. This task is made difficult by the all-paths execution model of symbolic evaluators, which defies both human intuition and standard profiling techniques. This paper presents symbolic profiling, a new approach to identifying and diagnosing performance bottlenecks in programs under symbolic evaluation. To help with diagnosis, we develop a catalog of common performance anti-patterns in solver-aided code. To locate these bottlenecks, we develop SymPro, a new profiling technique for symbolic evaluation. SymPro identifies bottlenecks by analyzing two implicit resources at the core of every symbolic evaluation engine: the symbolic heap and symbolic evaluation graph. These resources form a novel performance model of symbolic evaluation that is general (encompassing all forms of symbolic evaluation), explainable (providing programmers with a conceptual framework for understanding symbolic evaluation), and actionable (enabling precise localization of bottlenecks). Performant solver-aided code carefully manages the shape of these implicit structures; SymPro makes their evolution explicit to the programmer. To evaluate SymPro, we implement profilers for the Rosette solver-aided language and the Jalangi program analysis framework. Applying SymPro to 15 published solver-aided tools, we discover 8 previously undiagnosed performance issues. Repairing these issues improves performance by orders of magnitude, and our patches were accepted by the tools' developers. We also conduct a small user study with Rosette programmers, finding that SymPro helps them both understand what the symbolic evaluator is doing and identify performance issues they could not otherwise locate.

Program sketching with live bidirectional evaluation

“…In contrast to solver-based and symbolic execution techniques for partially evaluating programs with holes (e.g. [Bornholt and Torlak 2018;Feng et al 2017a;), live evaluation is a form of concrete evaluation, adapting ideas from contextual modal type theory [Nanevski et al 2008]. Omar et al [2019, §5] detail the relationship to related work on partial evaluation.…”

“…Compared to the aforementioned logic-based and other symbolic evaluation techniques (e.g. [Bornholt and Torlak 2018;Feng et al 2017a;), live bidirectional evaluation employs concrete evaluation to collect example constraints łgloballyž across multiple holes in the sketch.…”

Efficiently finding higher-order mutants

“…At control-flow decisions, both branches are explored under corresponding symbolic path conditions; afterward, state is merged again into conditional values to exploit sharing in subsequent statements. In a nutshell, variational execution can be considered as an extreme design choice among various forms of symbolic program evaluation [5,6,8,34,69] for finite domains, in which computations are maximally performed on concrete values, but Boolean symbolic values may distinguish between multiple concrete values in program state [3,53,75].…”

A Design and Verification Methodology for a TrustZone Trusted Execution Environment

“…However, the overall approaches are quite different. Serval uses Rosette [37] and relies on SMT solving, which allows for more automation. Our approach uses Coq, which can express richer properties that Serval cannot.…”