Interpreter-guided differential JIT compiler unit testing

Abstract

Modern language implementations using Virtual Machines feature diverse execution engines such as byte-code interpreters and machine-code dynamic translators, a.k.a. JIT compilers. Validating such engines requires not only validating each in isolation, but also that they are functionally equivalent. Tests should be duplicated for each execution engine exercising the same execution paths on each of them.In this paper we present a novel automated testing approach for virtual machines featuring byte-code interpreters. Our solution uses concolic meta-interpretation: it applies concolic testing to a byte-code interpreter to explore all possible execution interpreter paths and obtain a list of concrete values that explore such paths. We then use such values to apply differential testing on the VM interpreter and JIT compiler. This solution is based on two insights: (1) both the interpreter and compiler implement the same language semantics and (2) interpreters are simple executable specifications of those semantics and thus promising targets to (meta-) interpretation using concolic testing. We validated it on 4 different compilers of the open-source Pharo Virtual Machine and found 468 differences between them, produced by 91 different causes, organized in 6 different categories.

A Generative and Mutational Approach for Synthesizing Bug-Exposing Test Cases to Guide Compiler Fuzzing

“…The subsequent works [17,38,51,60,66,69,84] further improve code coverage for domain-specific testing by mutating the seed programs. Poloto et al [67] proposed an interpreter-guided unit testing solution on the JIT compiler. It employs concolic testing to explore all possible execution paths and the corresponding values of an interpreter and uses these concrete values to implement differential unit testing on multiple JIT compilers.…”

Heap Fuzzing: Automatic Garbage Collection Testing with Expert-Guided Random Events

“…Lately, several works have explored the path of program fuzzing and differential testing. Recent work explores the generation of test cases for JIT compilers using available interpreters [24]. Other solutions make use of differential testing between different Virtual Machines for a single language.…”

“…Existing fuzzing techniques applied to virtual machines (VM) are aimed at testing (just in time) JIT compiler engines and propose generally the use of template programs that are mutated [6], [7], [17], [21], [23], [24], [36]. We argue however that such approaches, although shown suitable to find compilation errors, are not the most efficient at finding garbage collection bugs.…”