FAST: Formal specification driven test harness generation

Abstract

Full coverage testing is commonly perceived as a mission impossible because software is more complex than ever and produces vast space to cover. This paper introduces a novel approach which uses ACSL formal specifications to define and reach test coverage, especially in the sense of data coverage. Based on this approach, we create a tool chain named FAST which can automatically generate test harness code and verify program's correctness, turning formal specification and static verification into coverage definition and dynamic testing. FAST ensures completeness of test coverage and result checking by leveraging the formal specifications. We have applied this methodology and tool chain to a real-world mission critical software project that requires high quality standard. Our practice shows using FAST detects extra code bugs that escape from other validation methods such as manually-written tests and random/fuzz tests. It also costs much less human efforts with higher bug detection rate and higher code and data coverage.

Exposing Non-Atomic Methods of Concurrent Objects

“…ese works are typically focused on one particular domain of program properties. For instance, some works automatically generate functional-conformance tests from given formal speci cations (Gong et al 2012), small litmus-test programs which exhibit non-sequentially-consistent memory behaviors (Alglave et al 2011;Mador-Haim et al 2011), or small programs which exhibit bugs in compiler optimizations (Chen et al 2013). e most-closely related work to ours also generates concurrent data structure test harnesses (Burckhardt et al 2010a); given a set of I invocations with xed arguments, and parameters T , S, N respectively for the number T of invocation sequences (threads), the number S of invocations per sequence, and the total number N of generated harnesses, they generate N combinations of T length-S invocation sequences using invocations I .…”