A comprehensive study of deep learning compiler bugs

Shen, Qingchao; Ma, Haoyang; Chen, Junjie; Tian, Yongqiang; Cheung, Shing-Chi; Chen, Xiang

doi:10.1145/3468264.3468591

Cited by 76 publications

(38 citation statements)

References 69 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…(1) Distributed DL testing. Existing DL testing work focuses on testing training data [32], DL frameworks [55], DL compilers [58], and DL models (such as their correctness, robustness, and fairness) [50,54], etc. To the best of our knowledge, there is little work that detects faults (e.g., misuse of multiple devices and misconfiguration of communication.)…”

Section: Discussionmentioning

confidence: 99%

Demystifying Developers' Issues in Distributed Training of Deep Learning Software

Liu¹,

Gu²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

Deep learning (DL) has been pervasive in a wide spectrum of nowadays software systems and applications. The rich features of these DL based software applications (i.e., DL software) usually rely on powerful DL models. To train powerful DL models with large datasets efficiently, it has been a common practice for developers to parallelize and distribute the computation and memory over multiple devices in the training process, which is known as distributed training. However, existing efforts in the software engineering (SE) research community mainly focus on issues in the general process of training DL models. In contrast, to the best of our knowledge, issues that developers encounter in distributed training have never been well studied. Given the surging importance of distributed training in the current practice of developing DL software, this paper fills in the knowledge gap and presents the first comprehensive study on developers' issues in distributed training. To this end, we extract and analyze 1,054 real-world developers' issues in distributed training from Stack Overflow and GitHub, two commonly used data sources for studying software issues. We construct a fine-grained taxonomy consisting of 30 categories regarding the fault symptoms and summarize common fix patterns for different symptoms. Based on the results, we suggest actionable implications and research avenues that can potentially facilitate the future development of distributed training.

show abstract

Section: Discussionmentioning

confidence: 99%

Demystifying Developers' Issues in Distributed Training of Deep Learning Software

Liu¹,

Gu²,

Chen³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In the study, we aim to investigate the characteristics of DL framework bugs, and thus we collected closed and merged pull requests that are responsible to fix bugs from the corresponding GitHub repositories following the existing work [16,24,38]. On the one hand, bugs involved in these pull requests have been accepted and fixed by developers; On the other hand, these pull requests tend to contain more comprehensive information, e.g., code changes, links to related issues, and discussions among developers, which is helpful to understand the bugs.…”

Section: Methodology 31 Data Collectionmentioning

confidence: 99%

“…Hence, we further identified bug-fixing pull requests through keyword searching in the tags and titles of pull requests. Following the existing work [16,24,38], we adopted several bug-relevant keywords, including fix, defect, error, bug, issue, mistake, correct, fault, and flaw.…”

Section: Methodology 31 Data Collectionmentioning

confidence: 99%

Toward Understanding Deep Learning Framework Bugs

Chen¹,

Liang²,

Shen³

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such wide effect demonstrates the necessity and importance of guaranteeing DL frameworks' quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating to design effective bug detection and debugging approaches. Hence, in this work we conduct the most large-scale study on 800 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with 5 components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques and developers' efforts on fixing those bugs, we obtain 14 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing and debugging practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging.

show abstract

“…To put these results in context, we compare against a study of bugs in deep learning compilers [Shen et al 2021]. Those bugs also most commonly manifest through crashes and incorrect outputs, but bugs in quantum computing platforms cause fewer crashes (41.2% vs. 59.37%) and more incorrect outputs (34.5% vs. 26.26%) than bugs in deep learning compilers.…”

Section: Rq3: How Do the Bugs Manifest?mentioning

confidence: 99%

“…Most closely related to our work are studies of bugs in widely used frameworks and platforms. For example, Chou et al [2001] study bugs in operating systems, while more recent work studies bugs in compilers [Sun et al 2016] and deep learning libraries [Islam et al 2019;Shen et al 2021].…”

Section: Studies Of Other Kinds Of Bugsmentioning

confidence: 99%

Bugs in Quantum Computing Platforms: An Empirical Study

Paltenghi¹,

Pradel²

2021

Preprint

View full text Add to dashboard Cite

The interest in quantum computing is growing, and with it, the importance of software platforms to develop quantum programs. Ensuring the correctness of such platforms is important, and it requires a thorough understanding of the bugs they typically suffer from. To address this need, this paper presents the first in-depth study of bugs in quantum computing platforms. We gather and inspect a set of 223 real-world bugs from 18 open-source quantum computing platforms. Our study shows that a significant fraction of these bugs (39.9%) are quantum-specific, calling for dedicated approaches to prevent and find them. The bugs are spread across various components, but quantum-specific bugs occur particularly often in components that represent, compile, and optimize quantum programming abstractions. Many quantum-specific bugs manifest through unexpected outputs, rather than more obvious signs of misbehavior, such as crashes. Finally, we present a hierarchy of recurrent bug patterns, including ten novel, quantum-specific patterns. Our findings not only show the importance and prevalence bugs in quantum computing platforms, but they help developers to avoid common mistakes and tool builders to tackle the challenge of preventing, finding, and fixing these bugs.

show abstract

A comprehensive study of deep learning compiler bugs

Cited by 76 publications

References 69 publications

Demystifying Developers' Issues in Distributed Training of Deep Learning Software

Demystifying Developers' Issues in Distributed Training of Deep Learning Software

Toward Understanding Deep Learning Framework Bugs

Bugs in Quantum Computing Platforms: An Empirical Study

Contact Info

Product

Resources

About