Metaheuristic search techniques have been extensively used to automate the process of generating test cases, and thus providing solutions for a more cost-effective testing process. This approach to test automation, often coined "Search-based Software Testing" (SBST), has been used for a wide variety of test case generation purposes. Since SBST techniques are heuristic by nature, they must be empirically investigated in terms of how costly and effective they are at reaching their test objectives and whether they scale up to realistic development artifacts. However, approaches to empirically study SBST techniques have shown wide variation in the literature. This paper presents the results of a systematic, comprehensive review that aims at characterizing how empirical studies have been designed to investigate SBST cost-effectiveness and what empirical evidence is available in the literature regarding SBST cost-effectiveness and scalability. We also provide a framework that drives the data collection process of this systematic review and can be the starting point of guidelines on how SBST techniques can be empirically assessed. The intent is to aid future researchers doing empirical studies in SBST by providing an unbiased view of the body of empirical evidence and by guiding them in performing well-designed and executed empirical studies.

No abstract

Critical systems such as those found in the avionics, automotive, maritime, and energy domains are often subject to a formal process known as certification. The goal of certification is to ensure that such systems will operate safely in the presence of known hazards, and without posing undue risks to the users, the public, or the environment. Certification bodies examine such systems based on evidence that the system suppliers provide, to ensure that the relevant safety risks have been sufficiently mitigated. Typically, generic safety standards set forth the general evidence requirements across different industry sectors, and then derived standards specialize the generic standards according to the needs of a specific industry sector. Regardless of whether a generic or sector-specific standard is being used, a key prerequisite for effective collection of evidence is that the supplier be aware of the requirements stipulated in the relevant standard and the evidence they require. This often proves to be a very challenging task because of the sheer size of the standards and the fact that the textual standards are amenable to subjective interpretation. Notably, suppliers find it hard to interpret the evidence requirements imposed by the safety standards within the domain of application; little support exists for recording, querying, and reporting evidence in a structured manner; and there is a general absence of guidelines on how the collected evidence supports the safety objectives. This thesis proposes the application of Model-Driven Engineering as an enabler for performing the various tasks related to safety evidence management. The position taken is that models should serve as the main source of certification information-documents, when needed, should be generated from models. Models are beneficial for the purpose of safety certification in many respects, most notably: (1) Models can be employed to clarify the expectations of safety standards and recommended practices, and develop concrete guidelines for system suppliers; (2) Models expressed in standard notations avoid the ambiguity and redundancy problems associated with text-based documentation; (3) Models provide an ideal vehicle for preserving traceability and the chain of evidence between hazards, requirements, design elements, implementation, and test cases; (4) Models can represent different levels of abstraction and an explicit mapping between the different levels; (5) Models present opportunities for partial or full automation of many laborious safety analysis tasks. The main contribution of this thesis is a model-driven process that enables the automated verification of compliance to standards based on evidence. Specifically, a UML profile is created, based on a conceptual model of a given standard, which provides a succinct and explicit interpretation of the underlying standard. The profile is augmented with constraints that help system suppliers with establishing a relationship between the concepts in the safety standard of interest and the concepts i...

No abstract