Eagle

Wang, Jiannan; Lutellier, Thibaud; Qian, Shangshu; Pham, Hūng Viêt; Tan, Lin

doi:10.1145/3510003.3510165

Cited by 16 publications

(6 citation statements)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To answer the RQs, we have performed an extensive study on PyTorch, TensorFlow, JAX, and OneFlow, whose details are shown in Table III. With 57.7K and 167K stats on GitHub, Py-Torch and TensorFlow are the two most popular DL libraries, and they are also widely studied in prior DL library testing work [30], [31], [41]. In addition, JAX [51] and OneFlow [52] are two emerging DL libraries, with 19.7K and 3.6K stars on GitHub.…”

Section: Methodsmentioning

confidence: 99%

“…Despite the recent advances in DL library testing [25], [27]- [31], [41], there is still limited work that can effectively test the crucial AD component for DL libraries. Therefore, this paper aims to build the first practical fuzzing technique specifically targeting AD in DL libraries.…”

Section: B Automatic Differentiationmentioning

confidence: 99%

“…Number of Detected Bugs. Following prior work on testing or fuzzing the DL libraries [25], [28]- [31], [41], [53], we report the number of bugs detected by ∇Fuzz and compared baselines. False Positive Rate.…”

Section: B Metricsmentioning

confidence: 99%

“…More specifically, FreeFuzz automatically tracks API inputs when running code mined from the open-source; additional mutations are performed to generate more inputs based on tracked seed API inputs. Another line of recent work designs other test oracles for DL libraries, e.g., DeepREL [53] automatically infers relational APIs (e.g., the APIs that should return the same values/statuses when given the same inputs) as the oracle to detect inconsistency bugs for DL libraries, while EAGLE [41] uses equivalent graphs to differentially test DL APIs. While effective, none of the existing API-level techniques targeted the crucial AD engines in DL libraries.…”

Section: Related Workmentioning

confidence: 99%

See 3 more Smart Citations

Fuzzing Automatic Differentiation in Deep-Learning Libraries

Yang¹,

Yinlin²,

Yao³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep learning (DL) has attracted wide attention and has been widely deployed in recent years. As a result, more and more research efforts have been dedicated to testing DL libraries and frameworks. However, existing work largely overlooked one crucial component of any DL system, automatic differentiation (AD), which is the basis for the recent development of DL. To this end, we propose ∇Fuzz, the first general and practical approach specifically targeting the critical AD component in DL libraries. Our key insight is that each DL library API can be abstracted into a function processing tensors/vectors, which can be differentially tested under various execution scenarios (for computing outputs/gradients with different implementations). We have implemented ∇Fuzz as a fully automated API-level fuzzer targeting AD in DL libraries, which utilizes differential testing on different execution scenarios to test both first-order and highorder gradients, and also includes automated filtering strategies to remove false positives caused by numerical instability. We have performed an extensive study on four of the most popular and actively-maintained DL libraries, PyTorch, TensorFlow, JAX, and OneFlow. The result shows that ∇Fuzz substantially outperforms state-of-the-art fuzzers in terms of both code coverage and bug detection. To date, ∇Fuzz has detected 173 bugs for the studied DL libraries, with 144 already confirmed by developers (117 of which are previously unknown bugs and 107 are related to AD). Remarkably, ∇Fuzz contributed 58.3% (7/12) of all high-priority AD bugs for PyTorch and JAX during a two-month period.None of the confirmed AD bugs were detected by existing fuzzers.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: B Automatic Differentiationmentioning

confidence: 99%

Section: B Metricsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

Fuzzing Automatic Differentiation in Deep-Learning Libraries

Yang¹,

Yinlin²,

Yao³

et al. 2023

Preprint

View full text Add to dashboard Cite

show abstract

“…Importantly, getting effective representation learning for mappings between the algorithms and code can help identify semantic clones that implement the same algorithmic steps. Such semantic clones are known to be very valuable for detecting bugs [33,18], generating test oracles and performing differential testings [5,9], fixing and improving programs [31], designing APIs and optimizing code [8], and providing data and downstream tasks for evaluating deep learning-based source code modeling tools [41,34,37,36,14] are mainly dependent on getting better representation learning. The mapping from pseudo code to source code can also provide insights on how to use pseudo code to synthesize the programs [21].…”

Section: Introductionmentioning

confidence: 99%

Representation learning for algorithms implementation search using pseudo code

Sium

View full text Add to dashboard Cite

CHAPTER 1. BACKGROUNDThis chapter introduces the importance of efficient and effective representation learning of a program graph, down stream task code search for an algorithm implementation, provides an overview of the different code search techniques, introduces our approach of code search for an algorithm. Next, we discuss related work focused for our focus area. In 1.2.1, we will discuss graph representation learning in general and specifically for programming language.

show abstract

Multigranularity Coverage Criteria for Deep Learning Libraries

Shi,

Zheng,

Yin

et al. 2024

Software Testing Verif & Rel

View full text Add to dashboard Cite

Deep learning (DL) systems are becoming increasingly widely used in safety domains such as self‐driving cars and unmanned aerial vehicles, which arouse natural concerns about their trustworthiness. Underlying DL libraries used in the construction and execution of DL models are involved in the testing processes of DL systems. Therefore, bugs in DL libraries can inevitably cause unexpected behaviours in DL systems. The internal structures of DL libraries are described as APIs with different functionalities, and DL libraries offer model developers access to DL techniques with various API parameter settings. The above characteristics of DL libraries reveal that existing DL coverage criteria are not designed specifically for DL libraries, and traditional software coverage criteria do not apply to DL libraries either. The paper introduces the first set of coverage criteria specifically designed for the systematic measurement of DL libraries across various granularities. APIs, as the fundamental components of DL libraries, are used to define coverage criteria gauging testing adequacy by thoroughly considering their invocation, implementation, parameter quantities and parameter attributes. Furthermore, some properties depicting relations between coverage criteria are investigated. Experiments on the effectiveness of the proposed coverage criteria and comparative analysis are conducted by interval estimate and hypothesis testing techniques for APIs in two well‐known DL libraries. The experimental results demonstrate that the proposed coverage criteria are effective in measuring the test adequacy of DL libraries, and they can be used for the quantitative analysis of test model quality in DL libraries.

show abstract

Eagle

Cited by 16 publications

References 32 publications

Fuzzing Automatic Differentiation in Deep-Learning Libraries

Fuzzing Automatic Differentiation in Deep-Learning Libraries

Representation learning for algorithms implementation search using pseudo code

Multigranularity Coverage Criteria for Deep Learning Libraries

Contact Info

Product

Resources

About