Measuring Test Data Uniformity in Acceptance Tests for the FitNesse and Gherkin Notations

Abstract

This paper presents two metrics designed to measure the data uniformity of acceptance tests in FitNesse and Gherkin notations. The objective is to measure the data uniformity of acceptance tests in order to identify projects with lots of random and meaningless data. Random data in acceptance tests hinder communication between stakeholders and increase the volume of glue code. The main contribution of this paper is the implementation of the proposed metrics. This paper also evaluates the uniformity of test data from several FitNesse and Gherkin projects found on GitHub, as a means to verify if the metrics are applicable. First, the metrics were applied to 18 FitNesse project repositories and 18 Gherkin project repositories. The measurements taken from these repositories were used to present cases of irregular and uniform test data. Then, we have compared the notations from FitNesse and Gherkin in terms of projects and features. In terms of projects, no significant difference was observed, that is, FitNesse projects have a level of uniformity similar to Gherkin projects. However, in terms of features and test documents, there was a significant difference. The uniformity scores of FitNesse and Gherkin features are 0.16 and 0.26, respectively. These uniformity scores are very low, which means that test data for both notations are very irregular. Thus, we can infer that test data are more irregular in FitNesse features than in Gherkin features. The evaluation also shows that 28 of 36 projects (78%) did not reach the minimum recommended measure, i.e., 0.45 of test data uniformity. In general, we can observe that there are still many challenges in improving the quality of acceptance tests, especially in relation to the uniformity of test data.

An empirical study to compare three web test automation approaches: NLP‐based, programmable, and capture&replay

“…Finally, both the works presented by Marchetto et al 43 and Longo et al 44 evaluate the adoption of Fitnesse ¶ ¶ ¶ ¶ and fit tables 45 as tools/ techniques to define acceptance tests in constrained natural languages. In particular, while Marchetto et al 43 compares fit tables for traditional systems and web-specific fit tables in maintenance tasks, Longo et al 44 compares Fitnesse and Gherkin projects from the point of view of test data uniformity. Uniform test data, as explained by the authors, are expressions that are common to various test documents.…”

Is NLP-based Test Automation Cheaper Than Programmable and Capture &Replay?

“…Fitnesse is another structured quasi-natural language adopted for specifying NL-based acceptance test cases. Both Marchetto et al [15] and Longo et al [13] evaluate the adoption of Fitnesse. While Marchetto et al [15] compare Fitnesse with programmable acceptance test cases, Longo et al [13] compares the adoption of Fitnesse and Gherkin for writing acceptance test cases.…”

“…OIM 12 is an inventory management that implements transactions management, raw material management, batch, supplier, items, categories, and storage management. The application has been mainly developed in PHP by using AppGini 13 , a web-database framework for applications building.…”

“…https://bigprof.com/appgini/applications/online-inventory-manager (last access: February 2022)13 https://appgini.en.softonic.com/…”