Preventing data errors with continuous testing

Abstract

Today, software systems that rely on data are ubiquitous, and ensuring the data's quality is an increasingly important challenge as data errors result in annual multi-billion dollar losses. While software debugging and testing have received heavy research attention, less effort has been devoted to data debugging: identifying system errors caused by well-formed but incorrect data. We present continuous data testing (CDT), a low-overhead, delay-free technique that quickly identifies likely data errors. CDT continuously executes domain-specific test queries; when a test fails, CDT unobtrusively warns the user or administrator. We implement CDT in the CON-TEST prototype for the PostgreSQL database management system. A feasibility user study with 96 humans shows that CONTEST was extremely effective in a setting with a data entry application at guarding against data errors: With CONTEST, users corrected 98.4% of their errors, as opposed to 40.2% without, even when we injected 40% false positives into CONTEST's output. Further, when using CONTEST, users corrected data entry errors 3.2 times faster than when using state-of-the-art methods.

Fairness testing: testing software for discrimination

“…For software systems that use data management, such provenance-based reasoning may aid testing for causal relationships between input characteristics and outputs. Our prior work on testing software that relies on data management systems has focused on data errors [59,60], whereas this work focuses on testing fairness.…”

It’s all in the eye of beholder

“…As mentioned in the introductory section, there are two points of view on the Reinhart-Rogoff controversy present in the literature. On the one hand, the commentators underline the spreadsheet error and other "flaws" committed by RR and advise replicating economic research (Clemens 2015;Muslu et al 2015;Dafoe 2014;Grimson 2014;Reiss 2014;Stevenson and Wolfers 2013). On the other hand, there are voices of criticism from the perspective of the "economics of economics".…”

“…The case study based on the threshold hypothesis focuses on the broadly known case of the two inconsistent findings obtained by Reinhart and Rogoff (2010a) and Herndon, Ash and Pollin (2014) 2 . To the best of our knowledge, the voices committed to the methodology applied in both the journal literature (for instance Clemens 2015;Muslu et al 2015;Dafoe 2014;Grimson 2014;Reiss 2014) as well as the popular press (e.g. Stevenson and Wolfers 2013) focus on the necessity of replicating economic analyses, making databases publicly available and underline the spreadsheet error committed by RR.…”

Software fairness

“…For software systems that use data management, such provenance-based reasoning may aid testing for causal relationships between input attributes and outputs. Our prior work on testing software that relies on data management systems has focused on data errors [61,62], whereas this work focuses on testing fairness.…”