SCRAM: Simple Checks for Realtime Analysis of Model Training for Non-Expert ML Programmers

Schoop, Eldon; Huang, Forrest; Hartmann, Björn

doi:10.1145/3334480.3382879

Cited by 5 publications

(13 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We compared and contrasted three approaches [32,39,43] against our approach (DD). From Table 7, we found our approach detected more problems in the SGS dataset than AUTOTRAINER.…”

Section: Results Discussionmentioning

confidence: 99%

“…These behaviors indicate that the network will not achieve high performance. These symptoms are often caused by the incorrect selection of DNN hyperparameters [32], such as loss function, activation function for the last layer, learning rate, optimizer, or batch size.…”

Section: 24mentioning

confidence: 99%

“…This table shows procedures descriptions from [1, 10,16,32,43]. T|F indicates that the procedure returns True| False respectively.…”

Section: S8mentioning

confidence: 99%

“…L|H indicates the procedure return Low| High respectively. We borrowed these methods from existing literature [1, 10,16,32,43] large/small; and (3) the learning rate is very high/low; and (4) there exist incorrect activation functions.…”

Section: S8mentioning

confidence: 99%

“…Internal Threat: When implementing Algorithm 1, Decision Tree (Figure 2), and Tables 2 and 3, we used the parameters defined by prior works [1, 10,16,32,43]. These selected values may not work for some unseen examples.…”

Section: Threats To Validitymentioning

confidence: 99%

See 4 more Smart Citations

DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs

Wardat¹,

Cruz²,

Le³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep Neural Networks (DNNs) are used in a wide variety of applications. However, as in any software application, DNN-based apps are afflicted with bugs. Previous work observed that DNN bug fix patterns are different from traditional bug fix patterns. Furthermore, those buggy models are non-trivial to diagnose and fix due to inexplicit errors with several options to fix them. To support developers in locating and fixing bugs, we propose DeepDiagnosis, a novel debugging approach that localizes the faults, reports error symptoms and suggests fixes for DNN programs. In the first phase, our technique monitors a training model, periodically checking for eight types of error conditions. Then, in case of problems, it reports messages containing sufficient information to perform actionable repairs to the model. In the evaluation, we thoroughly examine 444 models -53 real-world from GitHub and Stack Overflow, and 391 curated by AUTOTRAINER. DeepDiagnosis provides superior accuracy when compared to UMLUAT and DeepLocalize. Our technique is faster than AUTOTRAINER for fault localization. The results show that our approach can support additional types of models, while state-of-the-art was only able to handle classification ones. Our technique was able to report bugs that do not manifest as numerical errors during training. Also, it can provide actionable insights for fix whereas DeepLocalize can only report faults that lead to numerical errors during training. DeepDiagnosis manifests the best capabilities of fault detection, bug localization, and symptoms identification when compared to other approaches. CCS CONCEPTS• Computing methodologies → Neural networks; • Software and its engineering → Software testing and debugging.

show abstract

“…We compared and contrasted three approaches [32,39,43] against our approach (DD). From Table 7, we found our approach detected more problems in the SGS dataset than AUTOTRAINER.…”

Section: Results Discussionmentioning

confidence: 99%

Section: 24mentioning

confidence: 99%

“…This table shows procedures descriptions from [1, 10,16,32,43]. T|F indicates that the procedure returns True| False respectively.…”

Section: S8mentioning

confidence: 99%

Section: S8mentioning

confidence: 99%

Section: Threats To Validitymentioning

confidence: 99%

See 3 more Smart Citations