ExAIS

Schumi, Richard; Sun, Jun

doi:10.1145/3510003.3510112

Cited by 2 publications

(10 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…These works are motivated by the fact that bugs in AI frameworks might potentially affect all AI applications that are built with such frameworks. Due to that and also due to the fact that AI frameworks can still suffer from severe bugs [45], it is important to thoroughly and systematically test them. Existing AI framework testing techniques can be categorized into the following groups: differential testing [40,49,55] and metamorphic testing [24,25,46].…”

Section: Ai Framework Testingmentioning

confidence: 99%

“…Our technique utilises an executable AI semantics called ExAIS [45] that is written in the logical programming language Prolog [38]. Prolog is a declarative language that relies on first order logic.…”

Section: Exaismentioning

confidence: 99%

“…Neural networks can have a long training time of days or weeks, which makes it cumbersome to evaluate potential ways of fixing a neural network. Often the error messages that occur during AI development can be unrelated to the actual issue that needs to be fixed [32], the error messages can be inconsistent, or there can even be hidden issues that do not produce error messages [45]. Another difficulty comes from the underlying AI frameworks such as TensorFlow or PyTorch.…”

Section: Introductionmentioning

confidence: 99%

“…Given such a graph, every connection can cause potential precondition violations, and identifying them can be cumbersome. Moreover, the debug information that is provided by AI frameworks can be imprecise or inconsistent [45]. Due to that, the process of developing a new model can be time-consuming and frustrating.…”

Section: Introductionmentioning

confidence: 99%

“…Our approach relies on an existing semantics called ExAIS [45] which defines the functionality of almost all TensorFlow layers in the logical programming language Prolog. ExAIS contains a number of preconditions that produce debug messages and enable the identification of model bugs.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Semantic-Based Neural Network Repair

Schumi

Sun

2023

Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

Self Cite

View full text Add to dashboard Cite

Recently, neural networks have spread into numerous fields including many safety-critical systems. Neural networks are built (and trained) by programming in frameworks such as TensorFlow and PyTorch. Developers apply a rich set of pre-defined layers to manually program neural networks or to automatically generate them (e.g., through AutoML). Composing neural networks with different layers is error-prone due to the non-trivial constraints that must be satisfied in order to use those layers. In this work, we propose an approach to automatically repair erroneous neural networks. The challenge is in identifying a minimal modification to the network so that it becomes valid. Modifying a layer might have cascading effects on subsequent layers and thus our approach must search recursively to identify a "globally" minimal modification. Our approach is based on an executable semantics of deep learning layers and focuses on four kinds of errors which are common in practice. We evaluate our approach for two usage scenarios, i.e., repairing automatically generated neural networks and manually written ones suffering from common model bugs. The results show that we are able to repair 100% of a set of randomly generated neural networks (which are produced with an existing AI framework testing approach) effectively and efficiently (with an average repair time of 21.08s) and 93.75% of a collection of real neural network bugs (with an average time of 3min 40s). CCS CONCEPTS• Software and its engineering → Formal software verification; • Computing methodologies → Artificial intelligence; • Theory of computation → Constraint and logic programming.

show abstract