Skoll: distributed continuous quality assurance

Abstract

Quality assurance (QA)

A model-based distributed continuous quality assurance process to enhance the quality of service of evolving performance-intensive software systems

“…In our previous efforts [1] we performed these steps manually, making numerous errors. Overall, it took around 48 hours of CPU time to run the ∼50,000 experimental tasks dictated by the experimental design.…”

“…Managing these variable platform aspects effec- In our initial Skoll approach, creating a benchmarking experiment to measure QoS properties required QA engineers to write (1) the header files, source code, that implement the functionality, (2) the configuration and script files that tune the underlying ORB and automate running tests and output generation, and (3) project build files (e.g., makefiles) required to generate the executable code. Our experience during our initial feasibility study [1] revealed how tedious and error-prone this process was since it required multiple manual steps to generate benchmarks, thereby impeding productivity and quality in the QA process.…”

“…QA process designers must determine (1) how tasks will be decomposed into subtasks, (2) on what basis and in what order subtasks will be allocated to clients, (3) how subtasks will be implemented to execute on a potentially wide set of client platforms, (4) how subtask results will be merged together and interpreted, (5) if and how should the process adapt on-the-fly based on incoming results, and (6) how the results of the overall process will be summarized and communicated to software developers. To support this process we've developed the following components and services for use by Skoll QA process designers (a comprehensive discussion appears in [1]): Configuration space model. A cornerstone of Skoll is a formal model of a QA process's configuration space, which captures all valid configurations for QA subtasks.…”

“…Our earlier publications [1] on Skoll described its structure and functionality and presented results from a feasibility study that applied Skoll tools and processes to ACE [2] and TAO [3], which are large (i.e., over two million SLOC) reusable middleware packages targeted at performance-intensive software for DRE systems. Our initial work focused largely on building the Skoll infrastructure, which consisted of the languages, algorithms, mechanisms, and analysis techniques that tested the functional correctness of reusable software and its application to end-user systems.…”

Community-based, collaborative testing and analysis

“…It could contain other sub-components, but such architectural information would not be used in this focus area. In our own research we have started by developing a system and approach currently called QUASI (Quality as a Service Infrastructure) [11,5,6,9]. QUASI's analytical cornerstone is a model of the design space that implicitly captures all configurations on which test jobs might run.…”