Differentially testing soundness and precision of program analyzers

Klinger, Christian; Christakis, Maria; Wüstholz, Valentin

doi:10.1145/3293882.3330553

Cited by 29 publications

(7 citation statements)

References 72 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Differential testing has been successfully applied to various domains, most notably compilers and runtime systems [16], [19], [17], [18], [20]. Following this success, differential testing has been applied to many other domains, from program analyzers, such as model checkers [41], debuggers [42], and symbolic execution engines [43], to probabilistic programming languages [44], and software libraries and services [45], [46], [47], [48]. Inspired by this work, we also employ differential testing for finding bugs in ORM systems.…”

Section: Related Workmentioning

confidence: 99%

Data-Oriented Differential Testing of Object-Relational Mapping Systems

Sotiropoulos

Chaliasos

Atlidakis

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

We introduce, what is to the best of our knowledge, the first approach for systematically testing Object-Relational Mapping (ORM) systems. Our approach leverages differential testing to establish a test oracle for ORM-specific bugs. Specifically, we first generate random relational database schemas, set up the respective databases, and then, we query these databases using the APIs of the ORM systems under test. To tackle the challenge that ORMs lack a common input language, we generate queries written in an abstract query language. These abstract queries are translated into concrete, executable ORM queries, which are ultimately used to differentially test the correctness of target implementations. The effectiveness of our method heavily relies on the data inserted to the underlying databases. Therefore, we employ a solver-based approach for producing targeted database records with respect to the constraints of the generated queries. We implement our approach as a tool, called CYNTHIA, which found 28 bugs in five popular ORM systems. The vast majority of these bugs are confirmed (25 / 28), more than half were fixed (20 / 28), and three were marked as release blockers by the corresponding developers.Index Terms-Object-Relational Mapping, Differential Testing, Automated Testing 1 from django.db import models 2 class Person(models.Model): 3 age = models.IntegerField() 4 name = models.CharField(max_length=20) 5 ... 6 p1 = Person(age=31, name="John") 7 p1.save() 8 p2 = Person.objects.get(age=32) 9 p2.delete() 1 expr = (1 + T.col) 2 squared = (expr * expr) 3 T.select(fn.sum(expr), fn.avg(squared)).all() 4 // Generated SQL

show abstract

Section: Related Workmentioning

confidence: 99%

Data-Oriented Differential Testing of Object-Relational Mapping Systems

Sotiropoulos

Chaliasos

Atlidakis

et al. 2021

2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

show abstract

“…Klinger et al [42] represent the first automated technique for deferentially testing the soundness and precision of program analysers. To evaluate the six well-known developed program analysers, the researchers used tens of thousands of automatically generated benchmarks.…”

Section: Survey Of Related Workmentioning

confidence: 99%

A mutation framework for evaluating security analysis tools in IoT applications

Alalfi

Parveen

Nazzal

2021

Software Testing Verif & Rel

View full text Add to dashboard Cite

With the growing and widespread use of Internet of Things (IoT) in our daily life, its security is becoming more crucial. To ensure information security, we require better security analysis tools for IoT applications. Hence, this paper presents an automated framework to evaluate taint-flow analysis tools in the domain of IoT applications. First, we propose a set of mutational operators tailored to evaluate three types of sensitivity analysis, flow, path and context sensitivity. Then we developed mutators to automatically generate mutants for those types. We demonstrated the framework on a subset of mutational operators to evaluate three taint-flow analysers, SaINT, Taint-Things and FlowsMiner. Our framework and experiments ranked the taint analysis tools according to precision and recall as follows: Taint-Things (99% recall, 100% precision), FlowsMiner (100% recall, 87.6% precision) and SaINT (100% recall, 56.8% precision). To the best of our knowledge, our framework is the first framework to address the need for evaluating taint-flow analysis tools and specifically those developed for IoT SmartThings applications.

show abstract

“…On the formal verification side, there have been some pen-and-paper proofs, such as that of the Astree analyzer [12], some automatic and interactive proofs, such as [16,44], and some verification efforts, which include [2,26,31]. Testing efforts for program analyzers include e.g., static analyzers [13,28,49,52], symbolic execution engines [27], refactoring engines [14], compilers [29,30,32,43,47,50], SMT solvers [3], among others. Most of these testing approaches use programs in the target language as test cases and and apply testing techniques like fuzzing (e.g., [3,27,50]) or differential testing [35], (e.g., [3,[27][28][29]50]).…”

Section: Related Workmentioning

confidence: 99%

Testing Your (Static Analysis) Truths

Casso

Morales

López-García

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Static analysis is nowadays an essential component of many software development toolsets. Despite some notorious successes in the validation of compilers, comparatively little work exists on the systematic validation of static analyzers, whose correctness and reliability is critical if they are to be inserted in production environments. Contributing factors may be the intrinsic difficulty of formally verifying code that is quite complex and of finding suitable oracles for testing it. In this paper, we propose a simple, automatic method for testing abstract interpretationbased static analyzers. Broadly, it consists in checking, over a suite of benchmarks, that the properties inferred statically are satisfied dynamically. The main advantage of our approach is its simplicity, which stems directly from framing it within the Ciao assertion-based validation framework, and its blended static/dynamic assertion checking approach. We show that in this setting, the analysis can be tested with little effort by combining the following components already present in the framework: the static analyzer, the assertion run-time checking mechanism, the random test case generator, and the unit-test framework . Together they compose a tool that can effectively discover and locate errors in the different components of the analysis framework. We apply our approach to test some of CiaoPP's analysis domains over a wide range of programs, successfully finding non-trivial, previously undetected bugs, with a low degree of effort.logic programming Partially funded by MICINN PID2019-108528RB-C21 ProCode and the Madrid P2018/TCS-4339 BLOQUES-CM program.

show abstract

Differentially testing soundness and precision of program analyzers

Cited by 29 publications

References 72 publications

Data-Oriented Differential Testing of Object-Relational Mapping Systems

Data-Oriented Differential Testing of Object-Relational Mapping Systems

A mutation framework for evaluating security analysis tools in IoT applications

Testing Your (Static Analysis) Truths

Contact Info

Product

Resources

About