Automatic Fault Detection for Deep Learning Programs Using Graph Transformations

Abstract

Nowadays, we are witnessing an increasing demand in both corporates and academia for exploiting Deep Learning ( DL ) to solve complex real-world problems. A DL program encodes the network structure of a desirable DL model and the process by which the model learns from the training dataset. Like any software, a DL program can be faulty, which implies substantial challenges of software quality assurance, especially in safety-critical domains. It is therefore crucial to equip DL development teams with efficient fault detection techniques and tools. In this article, we propose NeuraLint , a model-based fault detection approach for DL programs, using meta-modeling and graph transformations. First, we design a meta-model for DL programs that includes their base skeleton and fundamental properties. Then, we construct a graph-based verification process that covers 23 rules defined on top of the meta-model and implemented as graph transformations to detect faults and design inefficiencies in the generated models (i.e., instances of the meta-model). First, the proposed approach is evaluated by finding faults and design inefficiencies in 28 synthesized examples built from common problems reported in the literature. Then NeuraLint successfully finds 64 faults and design inefficiencies in 34 real-world DL programs extracted from Stack Overflow posts and GitHub repositories. The results show that NeuraLint effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5% and a precision of 100%. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures such as recurrent neural networks. Researchers can also expand our set of verification rules to cover more types of issues in DL programs.

Bugs in Machine Learning-based Systems: A Faultload Benchmark

“…Then, it suggests the best practices to improve the quality of ML components. Nikanjam et al [52] provided an automatic fault detection tool for DL programs namely NeuraLint that validates the DL programs by detecting faults and design inefficiencies in the implemented models. They identified 23 different rules using graph transformations to detect various types of bugs in DL programs.…”

“…Our filters eliminated all their mentioned bugs and we could not achieve any bug from their dataset. Nikanjam et al [52] delivered a public dataset with their automatic bug detection tool including 34 real bugs in DL programs, 26 from SO and 8 from GitHub. After checking their provided bugs using mentioned exclusion criteria, we added 8 of GitHub bugs and 10 SO ones to the benchmark.…”

“…It is also worth noting to mention that some SO bugs are repetead in different public datasets. As an example, post [66] exists in three public datasets [52,79,31].…”

“…In this case study, we compare two testing tools for ML-based systems. We selected two up-to-date testing tools published recently: NeuraLint [52] and DeepLocalize [79]. NeuraLint is a model-based automatic fault detection tools for DL programs.…”

“…As an example, Denchmark [37] does not provide enough information to reproduce and trigger bugs. Meanwhile, several studies on the testing of ML-based systems have used synthetic bugs for assessment [54,52] which may bias their evaluation by hiding potential weaknesses. Some others have also used a limited number of real bugs [79,62] that may not be representative of a thorough evaluation, implying an incorrect measure of the proposed approach's reliability.…”