Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences

Schulz, Henning; Okanović, Dušan; Hoorn, André van; Ferme, Vincenzo; Pautasso, Cesare

doi:10.1145/3297663.3309674

Cited by 20 publications

(14 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Measurement of performance metrics under typical or stress test execution conditions, which involve both workload and platform configuration aspects (Menasc'e 2002;Hill, Schmidt, Edmondson & Gokhale 2009;Apte et al 2017;Michael et al 2017;Jindal et al 2019), detection of performance-related issues such as functional problems or violations of performance requirements emerging under certain workload or resource configuration conditions (Briand et al 2005;Zhang et al 2011;Ayala-Rivera et al 2018;Schulz et al 2019) are common objectives of different types of performance testing.…”

Section: Related Workmentioning

confidence: 99%

“…Behavior-driven declarative techniques. Using a Domain Specific Language (DSL) to provide declarative goal-oriented specifications of performance tests and model-driven execution frameworks for automated execution of the tests (Ferme and Pautasso 2018;Ferme and Pautasso 2017;Walter et al 2016), and using a high-level behavior-driven language inspired from Behavior-Driven Development (BDD) techniques to define test conditions (Schulz et al 2019) in combination with a declarative performance testing framework like BenchFlow (Ferme and Pautasso 2017) are examples of behavior-driven techniques for performance testing.…”

Section: Related Workmentioning

confidence: 99%

“…Testing approaches mainly raise issues of automated and efficient generation of test cases (test conditions) resulting in accomplishing the intended objective. Common approaches for generating the performance test cases such as using source code analysis (Zhang et al 2012), linear programs and evolutionary algorithms on performance models (Zhang and Cheung 2002;Gu and Ge 2009;Di Penta et al 2007) and UML models (Garousi 2010;Garousi 2008;Garousi et al 2008;Costa et al 2012;da Silveira et al 2011), using use case-based (Draheim et al 2006;Lutteroth and Weber 2008), and behavior-driven techniques (Schulz et al 2019;Ferme and Pautasso 2018;Ferme and Pautasso 2017;Walter et al 2016) mainly rely on source code or other artifacts, which might not always be available during the testing.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

et al. 2021

View full text Add to dashboard Cite

Test automation brings the potential to reduce costs and human effort, but several aspects of software testing remain challenging to automate. One such example is automated performance testing to find performance breaking points. Current approaches to tackle automated generation of performance test cases mainly involve using source code or system model analysis or use-case-based techniques. However, source code and system models might not always be available at testing time. On the other hand, if the optimal performance testing policy for the intended objective in a testing process instead could be learned by the testing system, then test automation without advanced performance models could be possible. Furthermore, the learned policy could later be reused for similar software systems under test, thus leading to higher test efficiency. We propose SaFReL, a self-adaptive fuzzy reinforcement learning-based performance testing framework. SaFReL learns the optimal policy to generate performance test cases through an initial learning phase, then reuses it during a transfer learning phase, while keeping the learning running and updating the policy in the long term. Through multiple experiments in a simulated performance testing setup, we demonstrate that our approach generates the target performance test cases for different programs more efficiently than a typical testing process and performs adaptively without access to source code and performance models.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

et al. 2021

View full text Add to dashboard Cite

show abstract

“…The characteristics of requirements elicitation are described in several ways, namely collaborative (stakeholder expertise), observational (system understanding and knowledge) and contextual (user requirements) [15][16], contextual (user requirements) [17][18]. The elicitation techniques are classified into brainstorming, workshops, Joint Application Development (JAD), group work, ethnography, introspection and user scenarios [19][20]. According to Wellsandt, the evaluation criteria are divided into two groups.…”

Section: Charateristics Of Requirements Elicitationmentioning

confidence: 99%

Comparison Elicitation Techniques in Furniture Manufacture Requirement

2021

View full text Add to dashboard Cite

The development of the industrial revolution 4.0 in manufacturing has changed conventional business processes to become automated, especially in the furniture sector. The constraints faced by the furniture manufacturing industry are that producers do not have a reporting tool that estimates how many products to sell, goods produced do not match their needs and stock errors. Therefore, the process of extracting needs is an important phase in the development of information systems because it determines the final outcome of the program. The research aims to define system requirements, so that it can help the decision-making process based on real-time data. This study describes the use of system planning techniques in the form of collaborative techniques, observational techniques and contextual techniques, which are combined with the Wellsandt evaluation criteria and measured using the low, medium and high scales. The results of this study are the acquisition of any business process areas that have a high level of difficulty to be prioritized and described in the form of a spider diagram. Implication of this research is that requirements can be used by developers to prioritize areas of business processes that must be improved in making decision support systems.

show abstract

“…We address these challenges by (i) automatically extracting and evolving performance tests using operational monitoring data and API information [14,17], (ii) the generation and selection of tailored tests based on current test concerns [12,13], as well as (iii) exploiting recommending test strategies suitable for testing in unreliable infrastructures [2,9].…”

Section: Continuous Testingmentioning

confidence: 99%

Performance Engineering for Microservices and Serverless Applications: The RADON Approach

Gias

Hoorn

Zhu

et al. 2020

Companion of the ACM/SPEC International Conference on Performance Engineering

Self Cite

View full text Add to dashboard Cite

Microservices and serverless are becoming integral parts of modern cloud-based applications. Tailored performance engineering is needed for assuring that the applications meet their requirements for quality attributes such as timeliness, resource efficiency, and elasticity. A novel DevOps-based framework for developing microservices and serverless applications is being developed in the RADON project. RADON contributes to performance engineering by including novel approaches for modeling, deployment optimization, testing, and runtime management. This paper summarizes the contents of our tutorial presented at the 11th ACM/SPEC International Conference on Performance Engineering (ICPE).

show abstract

Behavior-driven Load Testing Using Contextual Knowledge - Approach and Experiences

Cited by 20 publications

References 16 publications

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

An autonomous performance testing framework using self-adaptive fuzzy reinforcement learning

Comparison Elicitation Techniques in Furniture Manufacture Requirement

Performance Engineering for Microservices and Serverless Applications: The RADON Approach

Contact Info

Product

Resources

About