Improving Neural Network Verification through Spurious Region Guided Refinement

Yang, Pengfei; Li, Renjue; Li, Jianlin; Huang, Cheng-Chao; Wang, Jingyi; Sun, Jun; Bai, Xue; Zhang, Lijun

doi:10.48550/arxiv.2010.07722

Cited by 2 publications

(2 citation statements)

References 34 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Among the works, formal verification aims to prove that no adversarial examples exist in the neighborhood of a given input. Substantial progress has been made using approaches like abstract interpretation [63,83] and reachability analysis [67]. However, these formal verification techniques are in general computationally expensive and only scale to limited model structures and properties (e.g., local robustness [29]).…”

Section: Introductionmentioning

confidence: 99%

QuoTe : Quality-oriented Testing for Deep Learning Systems

Chen

Wang

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, i.e., given a property of test, defects of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the neuron coverage metrics, commonly used by most existing DL testing approaches, are not necessarily correlated with model quality (e.g., robustness, the most studied model property), and are also not an effective measurement on the confidence of the model quality after testing. In this work, we address this gap by proposing a novel testing framework called QuoTe (i.e., Qu ality- o riented Te sting). A key part of QuoTe is a quantitative measurement on 1) the value of each test case in enhancing the model property of interest (often via retraining), and 2) the convergence quality of the model property improvement. QuoTe utilizes the proposed metric to automatically select or generate valuable test cases for improving model quality. The proposed metric is also a lightweight yet strong indicator of how well the improvement converged. Extensive experiments on both image and tabular datasets with a variety of model architectures confirm the effectiveness and efficiency of QuoTe in improving DL model quality, i.e., robustness and fairness. As a generic quality-oriented testing framework, future adaptions can be made to other domains (e.g., text) as well as other model properties.

show abstract

Section: Introductionmentioning

confidence: 99%

QuoTe : Quality-oriented Testing for Deep Learning Systems

Chen

Wang

et al. 2023

ACM Trans. Softw. Eng. Methodol.

View full text Add to dashboard Cite

show abstract

“…Among them, formal verification aims to prove that no adversarial examples exist in the neighborhood of a given input. Substantial progress has been made using approaches like abstract interpretation [36], [51] and reachability analysis [40]. However, formal verification techniques are in general expensive and only scale to limited model structures and properties (e.g., local robustness [17]).…”

Section: Introductionmentioning

confidence: 99%

RobOT: Robustness-Oriented Testing for Deep Learning Systems

Wang

Chen

Sun

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Recently, there has been a significant growth of interest in applying software engineering techniques for the quality assurance of deep learning (DL) systems. One popular direction is deep learning testing, where adversarial examples (a.k.a. bugs) of DL systems are found either by fuzzing or guided search with the help of certain testing metrics. However, recent studies have revealed that the commonly used neuron coverage metrics by existing DL testing approaches are not correlated to model robustness. It is also not an effective measurement on the confidence of the model robustness after testing. In this work, we address this gap by proposing a novel testing framework called Robustness-Oriented T esting (RobOT). A key part of RobOT is a quantitative measurement on 1) the value of each test case in improving model robustness (often via retraining), and 2) the convergence quality of the model robustness improvement. RobOT utilizes the proposed metric to automatically generate test cases valuable for improving model robustness. The proposed metric is also a strong indicator on how well robustness improvement has converged through testing. Experiments on multiple benchmark datasets confirm the effectiveness and efficiency of RobOT in improving DL model robustness, with 67.02% increase on the adversarial robustness that is 50.65% higher than the state-of-the-art work DeepGini.

show abstract

Improving Neural Network Verification through Spurious Region Guided Refinement

Cited by 2 publications

References 34 publications

QuoTe : Quality-oriented Testing for Deep Learning Systems

QuoTe : Quality-oriented Testing for Deep Learning Systems

RobOT: Robustness-Oriented Testing for Deep Learning Systems

Contact Info

Product

Resources

About