Toward Understanding Deep Learning Framework Bugs

Abstract

DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such wide effect demonstrates the necessity and importance of guaranteeing DL frameworks' quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating to design effective bug detection and debugging approaches. Hence, in this work we conduct the most large-scale study on 800 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with 5 components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques and developers' efforts on fixing those bugs, we obtain 14 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing and debugging practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging.

Understanding Bugs in Multi-Language Deep Learning Frameworks

“…(2) In PyTorch, every indicator of MPL bug fixes is significantly greater than that of SPL bug fixes. (3) In TensorFlow, the LOCM, NOFM, and Entropy of MPL bug fixes are significantly larger than those of SPL bug fixes, respectively, while there are no significant differences between MPL bug fixes and SPL bug fixes on OT, NODP, and NOC.…”

COMET: Coverage-guided Model Generation For Deep Learning Library Testing

“…We also use Keras [11] to generate DL models for a fair comparison with LEMON [54] and Muffin [18]. Following existing works [18,19], we take the Keras documentation 6 as the reference to define the mutation space. In particular, we collect the information of all 59 Keras layer APIs (i.e., 𝑁 𝑙𝑎𝑦𝑒𝑟 = 59), including their possible datatypes, required input dimensions, and possible parameter values.…”

“…Table 6 presents the coverage results achieved by the baselines. Since collecting DL library coverage is costly and testing practices and bugs inside different DL libraries share a significant commonality [6,22,34,58], we used the branch and line coverage on TensorFlow's model construction and model execution modules as the representative for evaluating each technique's test coverage. According to Table 6, COMET outperforms the baselines on all five coverage criteria.…”

“…To alleviate the bias of measurement, our criteria for selecting the representative library to collect code coverage follow the practice of one recent JVM testing work [8]. Specifically, we choose TensorFlow as the representative because 1) it is the DL library with the highest GitHub stars and is the most popular one with the richest implementations, 2) all existing DL library testing techniques consider TensorFlow as their subject under test, and 3) the testing practices and bugs inside DL libraries share a significant commonality [6,22,34,58]. Therefore, the code coverage in TensorFlow can reflect DL library testing techniques' effectiveness in terms of test coverage.…”

“…This study indicates that inaccurate calculation result is the most common symptom of silent bugs. Chen et al [6] conduct a comprehensive study to facilitate a sufficient understanding of DL library bugs. Compared with previous studies, they took not only TensorFlow but also PyTorch, MXNet, and Deeplearning4j as the study subjects.…”