Exposing Bugs in JavaScript Engines through Test Transplantation and Differential Testing

Abstract

Context. JavaScript is a popular programming language today with several implementations competing for market dominance. Although a specification document and a conformance test suite exist to guide engine development, bugs occur and have important practical consequences. Implementing correct engines is challenging because the spec is intentionally incomplete and evolves frequently. Objective. This paper investigates the use of test transplantation and differential testing for revealing functional bugs in JavaScript engines. The former technique runs the regression test suite of a given engine on another engine. The latter technique fuzzes existing inputs and then compares the output produced by different engines with a differential oracle. Method. We conducted experiments with engines from five major players-Apple, Facebook, Google, Microsoft, and Mozilla-to assess the effectiveness of test transplantation and differential testing. Results. Our results indicate that both techniques revealed several bugs, many of which confirmed by developers. We reported 35 bugs with test transplantation (23 of these bugs confirmed and 19 fixed) and reported 24 bugs with differential testing (17 of these confirmed and 10 fixed). Results indicate that most of these bugs affected two engines-Apple's JSC and Microsoft's ChakraCore (24 and 26 bugs, respectively). To summarize, our results show that test transplantation and differential testing are easy to apply and very effective in finding bugs in complex software, such as JavaScript engines.

Interpreter-guided differential JIT compiler unit testing

“…Moreover, test generation has been explored in the case of native extensions for the JVM (i.e., JNI) [13]. Similar work appeared recently for JavaScript engines, applying differential testing with test transplantation [18] and compiler fuzzing [23,33]. Although we share with these approaches the goal of automatic test generation, these explore a more coarse test generation using and usually think of the VM as a black-box.…”

Heap Fuzzing: Automatic Garbage Collection Testing with Expert-Guided Random Events

“…Recent work explores cross-version comparison among a single JVM [38]. Similar work appeared recently for JavaScript engines using test transplantation [21] and compiler fuzzing [23], [36]. Although we share with these approaches the goal of automatic test generation, these explore JIT compiler testing and treat VMs as black boxes.…”

“…Existing fuzzing techniques applied to virtual machines (VM) are aimed at testing (just in time) JIT compiler engines and propose generally the use of template programs that are mutated [6], [7], [17], [21], [23], [24], [36]. We argue however that such approaches, although shown suitable to find compilation errors, are not the most efficient at finding garbage collection bugs.…”