Bifrost

Abstract

Live testing is used in the context of continuous delivery and deployment to test changes or new features in the production environment. This includes canary releases, dark launches, A/B tests, and gradual rollouts. Oftentimes, multiple of these live testing practices need to be combined (e.g., running an A/B test after a dark launch). Manually administering such multi-phase live testing strategies is a daunting task for developers or release engineers. In this paper, we introduce a formal model for multiphase live testing, and present Bifrost as a prototypical Node.js based middleware that allows developers to define and automatically enact complex live testing strategies. We extensively evaluate the runtime behavior of Bifrost in three rollout scenarios of a microservice-based case study application, and conclude that the performance overhead of our prototype is at or below 8 ms for most scenarios. Further, we show that more than 100 parallel strategies can be enacted even on cheap public cloud instances.

Test Plan Generation for Live Testing of Cloud Services

“…accuracy of the fault localization. The authors in Schermann et al (2016) propose a solution that uses canary testing and gradual rollouts to perfom tests in the production environment and improve error detection.…”

DevOps Test Süreç Geliştirmede Yeni Bir Model Önerisi

“… Sürekli Teslimat Testleri: Tamamı otomatik olarak geliştirilen fonksiyonel testlerdir. Ara yüz, servis ve veri tabanı gibi test nesnelerinin gereksinimlere uygunluğunu doğrulama amacını taşımaktadır [38].  Mikro Servis Testleri: DevOps ile birlikte monolitik mimari yerine kullanılan mikro servis mimarisinin test aktiviteleridir.…”

“…Hazırlanan anket soruları aşağıdaki gibidir:  Sürekli Entegrasyon Test Süreçleri: Otomasyon ile gerçekleştirilen birim, entegrasyon ve kontrat testlerini ifade etmektedir.  Sürekli Teslimat Test Süreçleri: Otomasyon ile geliştirilen fonksiyonel testleri ifade etmektedir [38].  Mikro Servis Test Aktiviteleri: Mikro servis testleri için uygulanan kontrat testlerini ve gerekli olan durumlarda sanallaştırılmış servislerin kullanılmasını ifade etmektedir [39].…”

An evaluation of open-source software microbenchmark suites for continuous performance assessment

“…As an alternative to invoke a full load test run with every build (which may be infeasible for many systems), existing techniques to judge the performance sensitivity of a commit could be utilized [20]. In industrial practice, and especially in the context of Software as a Service (SaaS) applications, application performance management [1,25] and partial rollouts (or canary releases) [44,48,50] are common approaches to unify CI with performance management. Fundamentally, these approaches eschew performance testing entirely, and instead rely on post-hoc identication of performance issues after deployment.…”