CiCUTS: Combining System Execution Modeling Tools with Continuous Integration Environments

Abstract

System execution modeling (SEM) tools provide an effective means to evaluate the quality of service (QoS) of enterprise distributed real-time and embedded (DRE) systems. SEM tools facilitate testing and resolving performance issues throughout the entire development life-cycle, rather than waiting until final system integration. SEM tools have not historically focused on effective testing. New techniques are therefore needed to help bridge the gap between the early integration capabilities of SEM tools and testing so developers can focus on resolving strategic integration and performance issues, as opposed to wrestling with tedious and error-prone low-level testing concerns.This paper provides two contributions to research on using SEM tools to address enterprise DRE system integration challenges. First, we evaluate several approaches for combining continuous integration environments with SEM tools and describe CiCUTS, which combines the CUTS SEM tool with the CruiseControl.NET continuous integration environment. Second, we present a case study that shows how CiCUTS helps reduce the time and effort required to manage and execute integration tests that evaluate QoS metrics for a representative DRE system from the domain of shipboard computing. The results of our case study show that CiCUTS helps developers and testers ensure the performance of an example enterprise DRE system is within its QoS specifications throughout development, instead of waiting until system integration time to evaluate QoS.

Unit Testing Non-functional Concerns of Component-based Distributed Systems

“…We therefore conclude that UNITE helped reduce the complexity of evaluating a non-functional concerns of a component-based distributed system. Moreover, unit tests can be automated and run continuously throughout the software development lifecycle of QED using CiCUTS [8], which integrates CUTS with continuous integration environments, such as CruiseControl (cruisecontrol. sourceforge.net).…”

“…In prior work [8], we showed how integrating CUTSbased SEM tools with continuous integtration environments provided a flexible solution for executing and managing component-based distributed system tests continuously throughout the development lifecycle. Our prior work also showed how simple log messages can capture arbitrary metrics of interest to evaluate non-functional concerns continuously throughout a system development.…”

Software paradigms, assessment types and non-functional requirements in model-based integration testing

“…Zhang et al [95] differ a bit from the standard coverage criteria as they propose them for real time and safety critical testing in extending standard flow criteria with time aspects. The approaches of Hill [38,39] use profiling techniques to capture performance metrics of executing systems. The values collected are for example the service time of events in a component, the number of received events, and the execution time of a databases query.…”

R&D challenges and solutions for mobile cyber-physical applications and supporting Internet services

“…SPOT then generates Java code for Android devices that allows developers to run and analyze their designs without the tedium of manual implementation. It also allows developers to perform continuous integration testing [14], which generated emulation code is incrementally replaced by actual cyber-physical application logic as the software evolves. This model-driven continuous integration process helps application developers increase the accuracy of their models throughout the software lifecycle.…”