An expert system for checking the correctness of memory systems using simulation and metamorphic testing

Cañizares, Pablo C.; Núñez, Alberto; Lara, Juan de

doi:10.1016/j.eswa.2019.04.070

Cited by 13 publications

(4 citation statements)

References 47 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Few methods or techniques can be used for software testing when the oracle problem exists. Among them, MT is a popular one in both the conventional software paradigm (Chen et al, 2020(Chen et al, , 2018 and the ML paradigm (Cañizares et al, 2019;Yin et al, 2022;Xie et al, 2011Xie et al, , 2020Xiao et al, 2022). In general, MT is a technique for generating test cases and verifying the test results, and it can be applied with or without the presence of a test oracle.…”

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

Evaluating the effectiveness of neuron coverage metrics: a metamorphic-testing approach

Zhou,

Poon,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) are now widely used in many sectors of our society. This phenomenon also means that if these DNNs contain faults, they will have profound adverse impacts on our daily lives. Thus, DNNs have to be comprehensively tested for ''correctness'' before they are released for use.Since such testing involves the use of a DNN test set, the comprehensiveness of this test set is of utmost importance. Until now, many researchers have proposed their own neuron-coverage (NC) metrics to measure the comprehensiveness of a DNN test set. However, their studies solely focused on those DNN testing scenarios with the presence of a test oracle. We observed that, in reality, there are many DNN testing scenarios where a test oracle does not exist and, therefore, the results of all previous studies may be inapplicable to these testing scenarios.Inspired by this observation, we have performed an empirical study to investigate the usefulness of some common and major NC metrics in terms of correlation analysis and invariability analysis. Our experiment results showed that, on the one hand, some NC metrics are useful measures of DNN test-set comprehensiveness (in terms of correlation analysis), but on the other hand, these metrics are not robust enough (in terms of invariability analysis).

show abstract

Section: Metamorphic Testing (Mt)mentioning

confidence: 99%

Evaluating the effectiveness of neuron coverage metrics: a metamorphic-testing approach

Zhou,

Poon,

Chen

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

“…During the last years, MeT [10,35] has been applied in a great variety of areas such as hardware validation [4,28], simulation [22,27], security [21], and machine learning [8,34]. The versatility provided by MeT allows us to analyse network communications in distributed systems, such as HPC clusters and cloud environments.…”

Section: Related Workmentioning

confidence: 99%

Validating communication network configurations in cloud and HPC systems using Metamorphic Testing

Cañizares

Pérez

Núñez

et al. 2022

2022 4th International Electronics Communication Conference (IECC)

Self Cite

View full text Add to dashboard Cite

During the last years, the fast evolution of computers and networks has led to the creation of a wide variety of services that have changed the way we live, like video streaming, on-line gaming and online shopping. These services are supported by complex systems, which require not only high computational power but high-speed and low-latency networks to fulfil the expected quality requirements. However, a misleading configuration in one of the thousand components that compose these systems may cause performance bottlenecks and functioning disruptions. Unfortunately, conventional testing methods are not adequate for checking these systems since, on many occasions, there does not exist a mechanism to determine if the behaviour of a system is the expected one. Fortunately, Metamorphic Testing is a valuable and promising testing technique that alleviates the two fundamental problems of testing: the oracle problem and the reliable test set problem.In this paper, we combine Metamorphic Testing and simulation techniques for validating communication network configurations in HPC systems. For this, we rely on a catalogue of Metamorphic Relations, based on network communications knowledge, for checking its correctness. In addition, we have conducted an experimental study for analysing the communication network of HPC systems. The results show that Metamorphic Testing is appropriate for checking the correctness of communication networks supported by complex topologies in HPC systems.CCS Concepts: • Networks → Network simulations; • Software and its engineering → Software testing and debugging.

show abstract

“…In this case, we executed up to 90,000 simulations to determine whether the MRs are valid when used with different cloud configurations. We measured the validity of the different MRs by using the percentage of test cases, for a correct cloud simulation, that satisfy each MR [61], [62], [63].…”

Section: Research Question 1: Are the Mrs Valid?mentioning

confidence: 99%

TEA-Cloud: A Formal Framework for Testing Cloud Computing Systems

et al. 2021

View full text Add to dashboard Cite

The validation of a cloud system can be complicated by the size of the system, the number of users that can concurrently request services, and the virtualisation used to give the illusion of using dedicated machines. Unfortunately, it is not feasible to use conventional testing methods with cloud systems. This paper proposes a framework, called TEA-Cloud, that integrates simulation with testing methods for validating cloud system designs. Testing is applied on both functional and non-functional aspects of the cloud, like performance and cost. The aim of the framework is to provide a complete methodology to help users to model both software and hardware parts of cloud systems and automatically test the validity of these clouds using a cost-effective approach. Metamorphic testing is used to overcome the lack of an oracle that checks whether the behaviour observed in testing is allowed. Metamorphic testing is based on metamorphic relations (MRs). We define three families of MRs, which target issues such as performance, resource provisioning and cost. TEA-Cloud was evaluated through an empirical study that used fault seeding (mutation) and ten MRs for testing different cloud configurations. The results were promising, with TEA-Cloud finding all seeded faults.

show abstract

An expert system for checking the correctness of memory systems using simulation and metamorphic testing

Cited by 13 publications

References 47 publications

Evaluating the effectiveness of neuron coverage metrics: a metamorphic-testing approach

Evaluating the effectiveness of neuron coverage metrics: a metamorphic-testing approach

Validating communication network configurations in cloud and HPC systems using Metamorphic Testing

TEA-Cloud: A Formal Framework for Testing Cloud Computing Systems

Contact Info

Product

Resources

About