Automating test oracles from restricted natural language agile requirements

Abstract

Manual testing of software requirements written in natural language for agile or any other methodology requires more time and human resources. This leaves the testing process error prone and time consuming. For satisfied end users with bug-free software delivered on time, there is a need to automate the test oracle process for natural language or informal requirements. The automation of the test oracle is relatively easier with formal requirements, but this task is difficult to achieve with natural language requirements. This study proposes an approach called Restricted Natural Language Agile Requirements Testing (ReNaLART) to automate the test oracle from restricted natural language agile requirements. For this purpose, it uses an existing user story template with some modifications for writing user stories. This helps in identifying test input and expected output for a user story. For comparison of expected and observed outputs it makes use of a regex pattern and string distance functions. It is capable of assigning different types of verdicts automatically depending upon the similarity/ dissimilarity between observed and expected outputs of user stories. ReNaLART is validated using several case studies of different domains, namely, OLX Pakistan, Mental Health Tests, McDelivery Pakistan, BlueStacks, Power Searching with Google, TensorFlow Playground, w3Schools 2018 offline and Touch'D. It revealed several faults in five of the above listed eight applications. Plus, the proposed test oracle on an average took 0.02 s for test data generation, expected output generation and verdict assignment. Both these facts show the fault revealing effectiveness and efficiency of ReNaLART. K E Y W O R D S automated test oracle, automated test-driven development, behaviour-driven development, restricted natural language, agile requirements 1 | INTRODUCTION Starting from the classic waterfall process model to the more recent agile process models, the focus of development has shifted from clearly gathered and well-documented requirements to handle creeping requirements with less focus on spending ample time on documenting requirements. Since the agile process caters for changing requirements, this poses a challenge for testers to test functionality corresponding to these changing requirements in an automated and efficient manner. As natural language is used widely as a tool in the software industry for documenting

Comparative Analysis of Text Mining and Clustering Techniques for Assessing Functional Dependency between Manual Test Cases

“…Domain experts then selected regression test cases matching an optimum threshold. Malik et al [21] present an approach of automated test oracle generation based on restricted natural language requirements. The observed and the expected output are compared using a regex pattern and several string distance functions.…”

Service quality in football tourism: an evaluation model based on online reviews and data envelopment analysis with linguistic distribution assessments

“…First, we leverage RegEx (Malik et al, 2021 ) and other normalization techniques to clean the text reviews by eliminating special characters (e.g., &, %, @, *, #, etc.) and unclarified numbers.…”

Conversion of User Story Scenarios to Python-Based Selenium Source Code for Automated Testing

“…Hence automation of the testing process is a possible solution to alleviate the problem. Automated testing is done using a software tool to test or check the software execution [15]. There are several benefits of implementing automated testing, such as faster execution than manual testing, more time for the testers, and the ability for the developers to create automated tests that often use the same programming language as the software product [16].…”