Interface Robustness Testing: Experience and Lessons Learned from the Ballista Project

Abstract

When the Ballista project started in 1996 as a 3-year DARPA-funded research project, the original goal was to create a Web-based testing service to identify robustness faults in software running on client computers via the Internet. Previous experience suggested that such tests would find interesting problems but it was unclear how to make robustness testing scalable to large interfaces. A major challenge was finding a way to test something as large and complex as an operating system (OS) application programming interface (API) without having to resort to labor-intensive manual test construction for each API function to be tested. In the end, a scalable approach was found and was successfully applied not only to operating system APIs but several other nonoperating system APIs as well.The robustness testing methodology Ballista is based upon using combinational tests of valid and invalid parameter values for system calls and functions. In each test case, a single software module under test (or MuT) is called once. An MuT can be a stand-alone program, function, system call, method, or any other software that can be invoked with a procedure call. (The term MuT is similar in meaning to the more recent term dependability benchmark target [DBench 2004].) In most cases, MuTs are calling points into an API. Each invocation of a test case determines whether a particular MuT provides robust exception handling when called with a particular set of parameter values. These parameter values, or test values, are drawn from a pool of normal and exceptional values based on the data type of each argument passed to the MuT. Each test value has an associated test object, which holds code to create and clean up the related system state for a test (for example, a file handle test object has code to create a file, return a file handle test value, and subsequently delete the file after the test case has been executed). A test case, therefore, consists of the name of the MuT and a tuple of test values that are passed as parameters Dependability Benchmarking for Computer Systems. Edited by Karama Kanoun and Lisa Spainhower 201

simFI: From single to simultaneous software fault injections

“…The authors do not differentiate between valid and invalid parameters that are combined it the tests and do not quantitatively analyze the impact of simultaneous injections, i.e., the combination of invalid inputs. However, they report that simultaneous injections have not had a significant impact on the detection probability of robustness vulnerabilities [34], [28], which contradicts our results. We see three possible reasons for this deviation, which we are planning to investigate in more depth in the future: (1) Ballista and our work target different interfaces of the OS (driver interface vs. API), (2) the quoted Ballista results were obtained more than a decade ago and may not hold for newer systems, and (3) the results may significantly differ due to the different fault models.…”

“…Service invocations at the kernel/driver interface are triggered by synthetic workloads designed to trigger executions of the targeted drivers. The applied failure detectors are capable of detecting four different classically known outcomes of an injection run [28]. Upon fault injection the system may respond with the following:…”

Automated system-level safety testing using constraint patterns for automotive operating systems

“…Our approach is closely related to an interface-level automated robustness testing [14,13]. Kropp et al implemented Ballista [14] to automate a robustness testing of POSIX API, but they typically focus on exercising exceptional parameters, rather than generating the sequence of API calls.…”

Challenges in Autonomous Vehicle Testing and Validation

“…Additionally, it has been shown that even very good designers often have blind spots and miss exceptional situations in comparatively simple software systems. [32] Fault injection and robustness testing are relatively mature technologies for assessing the performance of a system under exceptional conditions [33], and can help avoid designer and tester blind spots when testing exceptional condition responses. Traditional fault injection involves inserting bit flips into memory and communication networks.…”