Coverage-Directed Test Generation Automated by Machine Learning -- A Review

Ioannides, Charalambos; Eder, Kerstin

doi:10.1145/2071356.2071363

Cited by 52 publications

(29 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Machine learning approaches have been successfully applied to generate successively better tests to increase coverage across a wide range of microprocessor verification scenarios [24]. For example, GAs have been used to search for biases for pseudo-random test generators [39].…”

Section: Evolutionary Algorithmsmentioning

confidence: 99%

“…Test based methods trade off a nonexhaustive (reduced states and transitions covered) result for a more detailed implementation. Therefore, the goal of any test based method should be to cover as many states and transitions as possible, in order to provide the highest possible guarantees about the system in the absence of a proof [6,24]. To achieve this, we develop an approach to automatically improve test suitability for exposing MCM violations, and guide tests towards unexplored states and transitions.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

McVerSi: A test generation framework for fast memory consistency verification in simulation

Nagarajan

Elver

2016

2016 IEEE International Symposium on High Performance Computer Architecture (HPCA)

View full text Add to dashboard Cite

The memory consistency model (MCM), which formally specifies the behaviour of the memory system, is used by programmers to reason about parallel programs. It is imperative that hardware adheres to the promised MCM. For this reason, hardware designs must be verified against the specified MCM. One common way to do this is via executing tests, where specific threads of instruction sequences are generated and their executions are checked for adherence to the MCM. It would be extremely beneficial to execute such tests under simulation, i.e. when the functional design implementation of the hardware is being prototyped. Most prior verification methodologies, however, target post-silicon environments, which when applied under simulation would be too slow.We propose McVerSi, a test generation framework for fast MCM verification of a full-system design implementation under simulation. Our primary contribution is a Genetic Programming (GP) based approach to MCM test generation, which relies on a novel crossover function that prioritizes memory operations contributing to non-determinism, thereby increasing the probability of uncovering MCM bugs. To guide tests towards exercising as much logic as possible, the simulator's reported coverage is used as the fitness function. Furthermore, we increase test throughput by making the test workload simulation-aware. We evaluate our proposed framework using the Gem5 cycle accurate simulator in full-system mode with Ruby. We discover 2 new bugs due to the faulty interaction of the pipeline and the cache coherence protocol. Crucially, these bugs would not have been discovered through individual verification of the pipeline or the coherence protocol. We study 11 bugs in total. Our GP-based test generation approach finds all bugs consistently, therefore providing much higher guarantees compared to alternative approaches (pseudo-random test generation and litmus tests).

show abstract

Section: Evolutionary Algorithmsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

McVerSi: A test generation framework for fast memory consistency verification in simulation

Nagarajan

Elver

2016

2016 IEEE International Symposium on High Performance Computer Architecture (HPCA)

View full text Add to dashboard Cite

show abstract

“…To date, such relationship is constructed (or trained) using machine learning (see e.g. [12]). The effectiveness of them relies on the quality of the training samples.…”

Section: Related Workmentioning

confidence: 99%

“…[5,6,7,8,9,10,11,12,13]). This led to Coveragedriven Verification (CDV) which is well acknowledged and applied in industry.…”

Section: Introductionmentioning

confidence: 99%

Determining Cases of Scenarios to Improve Coverage in Simulation-based Verification

Yang

Wille

Drechsler

2014

Proceedings of the 27th Symposium on Integrated Circuits and Systems Design

View full text Add to dashboard Cite

Functional verification of complex designs is still dominated by simulation-based approaches. In particular, Coveragedriven Verification (CDV) is well acknowledged and applied in industry. Here, verification gaps in terms of inadequately checked scenarios are addressed and closed by generating and applying dedicated stimuli. In order to ensure a good coverage and, by this, a high verification quality, each scenario is supposed to become sufficiently triggered. However, the considered scenario may be triggered in several fashions and information about that is hardly available in the existing CDV approaches. In this work, we propose an approach which automatically derives this information. Examples and experimental evaluations illustrate how this improves coverage in simulation-based verification.

show abstract

“…Another way to improve the quality of randomly generated test sets is using coverage-directed generation [108,169]. This is an iterative and evolutionary approach, where at each step the fault coverage of a group of tests is evaluated by simulation, and at the next step the group is transformed in order to improve fault coverage and other desirable properties.…”

Section: Related Work: Testing Techniquesmentioning

confidence: 99%

Analysis and test of the effects of single event upsets affecting the configuration memory of SRAM-based FPGAs

Cassano

2014

2014 International Test Conference

View full text Add to dashboard Cite

SommarioI dispositivi FPGA con memoria di configurazione SRAM sono sempre più rilevanti in un grande numero di campi applicativi, dal contesto automobilistico a quello aerospaziale. Questi campi applicativi sono caratterizzati dalla presenza di radiazioni capaci di causare Single Event Upsets (SEUs) in dispositivi digitali. Tali guasti hanno effetti particolarmente dannosi sui sistemi implementati in tecnologia SRAM-based FPGA, in quanto sono in grado non solo di danneggiare temporaneamente il comportamento del sistema, cambiando il contenuto di flip-flop e memorie, ma anche di cambiare permanentemente la funzionalità implementata dal sistema stesso, cambiando il contenuto della memoria di configurazione. Il design di applicazioni safetycritical richiede l'utilizzo, prima possibile durante il flusso di progetto del sistema, di metodologie accurate per la valutazione della sensitività ai SEU del sistema stesso. Inoltre è necessario essere in grado di rilevare l'occorrenza di SEU durante il funzionamento del sistema. A questo scopo è necessario generare test patterns durante il progetto del sistema ed è poi necessario applicare tali test patterns agli input del sistema durante il suo funzionamento.In questa tesi descriviamo il progetto e l'implementazione di strumenti software utili al progettista di applicazioni safety-critical basati su tecnologia SRAM-based FP-GA per la valutazione della sensitività ai SEU del sistema e per la generazione di test pattern utili al rilevamento di SEU nella memoria di configurazione durante la vita del sistema. La caratteristica principale di questi strumenti è l'implementazione di un modello di SEU nei bit di configurazione che controllano le risorse logiche e di routing di un dispositivo FPGA che risulta essere molto più accurato rispetto ai classici modelli stuck-at ed open/short che sono in genere considerati nell'analisi di circuiti digitali. In tal modo gli strumenti proposti risultano essere molto più accurati rispetto a strumenti simili, sia accademici che commerciali, attualmente disponibili per l'analisi dei guasti in dispositivi digitali ma non specificamente sviluppati per dispositivi FPGA.In particolare tre strumenti sono stati progettati ed implementati: (i) ASSESS: Accurate Simulator of SEuS affecting the configuration memory of SRAM-based FPGAs, un simulatore di SEU nella memoria di configurazione di sistemi implementati in tecnologia SRAM-based FPGA, finalizzato a valutare la sensitività del sistema ai SEU prima possibile nel processo di sviluppo del sistema; (ii) UA 2 TPG: Untestability Analyzer and Automatic Test Pattern Generator for SEUs Affecting the Configuration Memory of SRAM-based FPGAs, uno strumento di analisi statica per l'identificazione dei SEU non testabili e per la generazione automatica di test patterns per il rilevamento del 100% dei SEU testabili; e (iii) GABES: Genetic Algorithm Based Environment for SEU Testing in SRAM-FPGAs, un ambiente basato su un algoritmo genetico per la generazione ed ottimizzazione di test patterns per il rilevamento di...

show abstract

Coverage-Directed Test Generation Automated by Machine Learning -- A Review

Cited by 52 publications

References 40 publications

McVerSi: A test generation framework for fast memory consistency verification in simulation

McVerSi: A test generation framework for fast memory consistency verification in simulation

Determining Cases of Scenarios to Improve Coverage in Simulation-based Verification

Analysis and test of the effects of single event upsets affecting the configuration memory of SRAM-based FPGAs

Contact Info

Product

Resources

About