We present a case study that illustrates the power of active learning for enabling the automated quality assurance of complex and distributed evolving systems. We illustrate how the development of the OCS, Springer Verlag's Online Conference System, is supported by continuous learning-based testing, that by its nature maintains the synchrony of the running application and the learned (test) model. The evolution of the test model clearly indicates which portions of the system remain stable and which are altered. Thus our approach includes classical regression testing and feature interaction detection. We show concretely how model checking, automata learning, and quantitative analysis concur with the holistic quality assurance of this product.Keywords-active learning, quality assurance, model-based testing.
I. MOTIVATIONIn a systems lifecycle, the effort spent for quality assurance and in particular testing often exceeds the development time. Already the combined testing effort prior to the first customer installation lies typically well above 20-30% of the overall costs, and this percentage gets higher with each new patch or upgrade. Still, far too many functional bugs escape the testing phases, and things get worse if one also considers non-functional properties like performance, which require specific (stress) test, or even user acceptance (of certain features), which can only be revealed via monitoring of the real user behavior. The speedup in releases and deployment advocated by many in industry [5] in the context of business critical applications will be realizable only, if adequate quality assurance mechanisms provide a variety of immediate and diagnosis-friendly feedback at the different stages of the lifecycle.The case study presented in this paper summarizes the experience we gained during the development, validation, actual operation, and use of the Online Conference Service (OCS), Springer Verlag's online manuscript submission and review system. Flexibility to add and adapt features is a must for the OCS, as it serves quite different scientific disciplines (besides Computer Science and Mathematics also various engineering disciplines etc.) all of them with their special requirements. Thus guaranteeing stability, security, and adequate performance is a challenging task which we address by a novel continuous quality assurance approach, which is in particular sensitive to the actual run-time behavior on its enterprise platform operated and maintained by Springer in the Netherlands.Backbone of this continuous quality assurance in the OCS life-cycle is (active) automata learning [1], here applied as a technology to infer behavioral models of the fully deployed enterprise-level system via systematic testing. We will discuss how this approach helps to identify and to resolve unwanted feature interactions [6], to reveal quantitative aspect that indicate performance problems, and to aggregate monitored user behavior along the perviously learned behavioral models. Besides applying advanced automata learning techn...