Snowtrail

Yan, Jiaqi; Jin, Qiuye; Jain, Shrainik; Viglas, Stratis D.; Lee, Allison

doi:10.1145/3209950.3209958

Cited by 19 publications

(3 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This demonstrates that the complex database states generated by QPG are beneficial not only to finding logic bugs, but also to other kinds of bugs. Although CockroachDB used the TLP oracle in their Continuous Integration (CI) process, 4 we still found 16 previously unknown bugs using QPG. For the new features in SQLite, QPG found 13 bugs in RIGHT JOIN, 2 bugs in FULL JOIN, and no bug in STRICT.…”

Section: Q1 New Bugsmentioning

confidence: 96%

“…Various techniques have been proposed to find logic bugs in DBMSs. Differential testing [47] is a general technique that compares the outputs of multiple systems for the same input to detect potential discrepancies indicating bugs; various approaches use it as a test oracle for finding logic bugs by using different DBMSs [16], [48], [49] or different versions of a DBMS [4], [50]. While such approaches have successfully found bugs, they are prone to false alarms due to differences in SQL dialects or expected differences between versions.…”

Section: Related Workmentioning

confidence: 99%

“…They are used in almost every computing device [1]- [3], thus any bug has a potentially severe consequence. Logic bugs, which refer to incorrect results returned by DBMSs, are a particularly challenging category of bugs to find as they silently compute an incorrect result-unlike, for example, crash bugs [4], [5], which cause the process to be terminated. Consider Listing 1, where the SELECT statement triggers a logic bug that causes the returned result to unexpectedly contain a record, while it should be empty.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Testing Database Engines via Query Plan Guidance

Ba,

Rigger

2023

2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)

View full text Add to dashboard Cite

Database systems are widely used to store and query data. Test oracles have been proposed to find logic bugs in such systems, that is, bugs that cause the database system to compute an incorrect result. To realize a fully automated testing approach, such test oracles are paired with a test case generation technique; a test case refers to a database state and a query on which the test oracle can be applied. In this work, we propose the concept of Query Plan Guidance (QPG) for guiding automated testing towards "interesting" test cases. SQL and other query languages are declarative. Thus, to execute a query, the database system translates every operator in the source language to one of the potentially many so-called physical operators that can be executed; the tree of physical operators is referred to as the query plan. Our intuition is that by steering testing towards exploring a variety of unique query plans, we also explore more interesting behaviors-some of which are potentially incorrect. To this end, we propose a mutation technique that gradually applies promising mutations to the database state, causing the DBMS to create potentially unseen query plans for subsequent queries. We applied our method to three mature, widely-used, and extensively-tested database systems-SQLite, TiDB, and CockroachDB-and found 53 unique, previously unknown bugs. Our method exercises 4.85-408.48× more unique query plans than a naive random generation method and 7.46× more than a code coverage guidance method. Since most database systemsincluding commercial ones-expose query plans to the user, we consider QPG a generally applicable, black-box approach and believe that the core idea could also be applied in other contexts (e.g., to measure the quality of a test suite).

show abstract

Section: Q1 New Bugsmentioning

confidence: 96%

Section: Related Workmentioning

confidence: 99%