How to Specify It!

Hughes, John

doi:10.1007/978-3-030-47147-7_4

Cited by 5 publications

(1 citation statement)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…More recently, MT has been increasingly gaining interest in classic AI fields for testing systems powered by machine learning, including: machine translation (Zhou and Sun, 2018;He et al, 2019), autonomous driving (Zhang et al, 2018;Zhou and Sun, 2019), and generic NLP (natural language processing) models (Ma et al, 2020;Ribeiro et al, 2020). MT can have comparable bug-revealing effectiveness to model-based testing, and hence is a useful alternative to test an implementation, especially in situations where a model is expensive to construct (Hughes, 2020).…”

Section: Introductionmentioning

confidence: 99%

Using Metamorphic Relations to Verify and Enhance Artcode Classification

Xu¹,

Towey²,

French³

et al. 2021

Preprint

View full text Add to dashboard Cite

Software testing is often hindered where it is impossible or impractical to determine the correctness of the behaviour or output of the software under test (SUT), a situation known as the oracle problem. An example of an area facing the oracle problem is automatic image classification, using machine learning to classify an input image as one of a set of predefined classes. An approach to software testing that alleviates the oracle problem is metamorphic testing (MT). While traditional software testing examines the correctness of individual test cases, MT instead examines the relations amongst multiple executions of test cases and their outputs. These relations are called metamorphic relations (MRs): if an MR is found to be violated, then a fault must exist in the SUT. This paper examines the problem of classifying images containing visually hidden markers called Artcodes, and applies MT to verify and enhance the trained classifiers. This paper further examines two MRs, Separation and Occlusion, and reports on their capability in verifying the image classification using one-way analysis of variance (ANOVA) in conjunction with three other statistical analysis methods: t-test (for unequal variances), Kruskal-Wallis test, and Dunnett's test. In addition to our previously-studied classifier, that used Random Forests, we introduce a new classifier that uses a support vector machine, and present its MRaugmented version. Experimental evaluations across a number of performance metrics show that the augmented classifiers can achieve better performance than non-augmented classifiers. This paper also analyses how the enhanced This work is an extension of an MET '18 paper (Xu et al., 2018), which was completed while the first author was a PhD student at University of Nottingham Ningbo China.

show abstract

Section: Introductionmentioning

confidence: 99%

Using Metamorphic Relations to Verify and Enhance Artcode Classification

Xu¹,

Towey²,

French³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

On the effectiveness of testing sentiment analysis systems with metamorphic testing

Jiang

Chen

Wang

2022

Information and Software Technology

View full text Add to dashboard Cite

Reflecting on Random Generation

Goldstein,

Frohlich,

Wang

et al. 2023

Proc. ACM Program. Lang.

View full text Add to dashboard Cite

Expert users of property-based testing often labor to craft random generators that encode detailed knowledge about what it means for a test input to be valid and interesting. Fortunately, the fruits of this labor can also be put to other uses. In the bidirectional programming literature, for example, generators have been repurposed as validity checkers, while Python's Hypothesis library uses the same structures for shrinking and mutating test inputs. To unify and generalize these uses and many others, we propose reflective generators, a new foundation for random data generators that can "reflect" on an input value to calculate the random choices that could have been made to produce it. Reflective generators combine ideas from two existing abstractions: free generators and partial monadic profunctors. They can be used to implement and enhance the aforementioned shrinking and mutation algorithms, generalizing them to work for any values that can be produced by the generator, not just ones for which a trace of the generator's execution is available. Beyond shrinking and mutation, reflective generators generalize a published algorithm for example-based generation, and they can also be used as checkers, partial value completers, and other kinds of test data producers.

show abstract

How to Specify It!

Cited by 5 publications

References 22 publications

Using Metamorphic Relations to Verify and Enhance Artcode Classification

Using Metamorphic Relations to Verify and Enhance Artcode Classification

On the effectiveness of testing sentiment analysis systems with metamorphic testing

Reflecting on Random Generation

Contact Info

Product

Resources

About