CanaryAdvisor: a statistical-based tool for canary testing (demo)

Abstract

Canary testing is an emerging technique that offers to minimize the risk of deploying a new version of software. It does so by slowly transferring load from the current to the new ("canary") version. As this ramp-up progresses, a human compares the performance and correctness of the two versions, and assesses whether to abort the canary version. For canary testing to be effective, a plethora of metrics must be analyzed, including CPU utilization and logged errors, across hundreds to thousands of machines. Performing this analysis manually is both time consuming and error prone.In this paper, we present CanaryAdvisor, a tool for automatic canary testing of cloud-based applications. CanaryAdvisor continuously monitors the deployed versions of an application and detects degradations in correctness, performance, and/or scalability. We describe our design and implementation of the CanaryAdvisor and outline open challenges.

Runtime Monitoring in Continuous Deployment by Differencing Execution Behavior Model

“…Execution logs have been extensively studied in such contexts as anomaly detection [4] [20], identification of software components [24], component behavior discovery [17], process mining [28], behavioral differencing [12], failure diagnosis [25], fault localization [30], invariant inference [5], and performance diagnosis [10] [26]. In this section, we focus on automatic analysis of execution logs.…”

“…Tarvo et al [26] automatically compare the quality of the canary version with the deployed version using a set of performance metrics such as CPU utilization and logged errors however, do not detect the differences in execution flow which is crucial for finding discrepancies. A set of techniques compare multiple versions of an application.…”

“…A continuous deployment model is crucial for service delivery business as it ensures that software services are always in a releasable state, and changes are incremental. To ensure high quality release in continuous deployment, the upcoming release is staged in production environment using such strategies as blue-green deployment [13], dark launches [9], canary release [13] [26] and shadow testing [23], and its performance is monitored [3][13] to quickly identify whether it is misbehaving [23] [26]. Fig.…”

Continuous experimentation for software developers

“…route) live user traffic on various levels (i.e., data center, server) for identifying and resolving (performance) bottlenecks across their application ecosystem. Tarvo et al [21] proposed a tool for automated canary testing incorporating data collection and analysis. Non-academic work includes open source tools such as Vamp 1 and Spinnaker 2 .…”

An empirical study on principles and practices of continuous delivery and deployment

“…Considering A/B testing, Tamburrelli and Margara [34] rephrase A/B testing as a search-based software engineering problem targeting automation by relying on aspect-oriented programming and genetic algorithms. Tarvo et al [36] built a tool for automated canary testing of cloud-based applications incorporating the automated collection and analysis of metrics using statistics.…”