An empirical study on TensorFlow program bugs

Zhang, Yuhao; Chen, Yifan; Cheung, Shing-Chi; Xiong, Yingfei; Zhang, Lu

doi:10.1145/3213846.3213866

Cited by 267 publications

(243 citation statements)

References 30 publications

Supporting

Mentioning

227

Contrasting

Unclassified

Order By: Relevance

“…Our study show that most of the deep learning bugs are Data Bugs and Logic Bugs [5], the primary root causes that cause the bugs are Structural Inefficiency (SI) and Incorrect Model Parameter (IPS) [5], most of the bugs happen in the Data Preparation stage of the deep learning pipeline. Our study also confirms some of the findings of Tensorflow conducted by Zhang et al [25]. We have also studied some antipatterns in the bugs to find whether there is any commonality in the code patterns that results in bugs.…”

Section: Introductionsupporting

confidence: 90%

“…A supervised pilot study and open coding schemes were used to identify the effects that are possible through these bugs. We have adapted the classification scheme of root causes and bug effects from [25] and added on top of that as found from the study of the posts. The third author studied the posts initially to finalize the classification scheme for bug types, root causes and effects.…”

Section: Classificationmentioning

confidence: 99%

See 1 more Smart Citation

A comprehensive study on deep learning bug characteristics

Islam

Nguyen

Pan

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

254

222

View full text Add to dashboard Cite

Deep learning has gained substantial popularity in recent years. Developers mainly rely on libraries and tools to add deep learning capabilities to their software. What kinds of bugs are frequently found in such software? What are the root causes of such bugs? What impacts do such bugs have? Which stages of deep learning pipeline are more bug prone? Are there any antipatterns? Understanding such characteristics of bugs in deep learning software has the potential to foster the development of better deep learning platforms, debugging mechanisms, development practices, and encourage the development of analysis and verification frameworks. Therefore, we study 2716 high-quality posts from Stack Overflow and 500 bug fix commits from Github about five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand the types of bugs, root causes of bugs, impacts of bugs, bug-prone stage of deep learning pipeline as well as whether there are some common antipatterns found in this buggy software. The key findings of our study include: data bug and logic bug are the most severe bug types in deep learning software appearing more than 48% of the times, major root causes of these bugs are Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) showing up more than 43% of the times. We have also found that the bugs in the usage of deep learning libraries have some common antipatterns that lead to a strong correlation of bug types among the libraries.

show abstract

Section: Introductionsupporting

confidence: 90%

Section: Classificationmentioning

confidence: 99%

A comprehensive study on deep learning bug characteristics

Islam

Nguyen

Pan

et al. 2019

Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of

254

222

View full text Add to dashboard Cite

show abstract

“…As for the second category (related to the characterization of particular root causes of bugs), Aslam et al [8] defined a classification of security faults in the Unix operating system. More recently, Zhang et al [95] analyzed the symptoms and root causes of 175 TensorFlow coding bugs from GitHub issues and StackOverflow questions. As a result, they proposed a number of challenges for their detection and localization.…”

Section: Bug Classification Schemasmentioning

confidence: 99%

Not all bugs are the same: Understanding, characterizing, and classifying bug types

Catolino

Palomba

Zaidman

et al. 2019

Journal of Systems and Software

View full text Add to dashboard Cite

“…There has been a study on the popularity of prevalence of correctness problems among all the reported ML bugs: Zhang et al [152] studied 175 Tensorflow bug reports from StackOverflow QA (Question and Answer) pages and from Github projects. Among the 175 bugs, 40 of them concern poor correctness.…”

Section: Correctnessmentioning

confidence: 99%

“…The study of Zhang et al [152] indicates that the most common learning program bug is due to the change of TensorFlow API when the implementation has not been updated accordingly. Additionally, 23.9% (38 in 159) of the bugs from ML projects in their study built based on TensorFlow arise from problems in the learning program.…”

Section: Bug Detection In Learning Programmentioning

confidence: 99%

Machine Learning Testing: Survey, Landscapes and Horizons

Zhang

Harman

Ма

et al. 2022

IIEEE Trans. Software Eng.

532

318

View full text Add to dashboard Cite

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 128 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing. Index Terms-machine learning, software testing, deep neural network, ! • Jie M. Zhang and Mark Harman are with CREST, University College London, United Kingdom. Mark Harman is also with Facebook London.

show abstract

An empirical study on TensorFlow program bugs

Cited by 267 publications

References 30 publications

A comprehensive study on deep learning bug characteristics

A comprehensive study on deep learning bug characteristics

Not all bugs are the same: Understanding, characterizing, and classifying bug types

Machine Learning Testing: Survey, Landscapes and Horizons

Contact Info

Product

Resources

About