Automatic Editing for Business Surveys: An Assessment of Selected Algorithms

Abstract

Statistical offices are responsible for publishing accurate statistical information about many different aspects of society. This task is complicated considerably by the fact that data collected by statistical offices generally contain errors. These errors have to be corrected before reliable statistical information can be published. This correction process is referred to as statistical data editing. Traditionally, data editing was mainly an interactive activity with the aim to correct all data in every detail. For that reason the data editing process was both expensive and time-consuming. To improve the efficiency of the editing process it can be partly automated. One often divides the statistical data editing process into the error localisation step and the imputation step. In this article we restrict ourselves to discussing the former step, and provide an assessment, based on personal experience, of several selected algorithms for automatically solving the error localisation problem for numerical (continuous) data. Our article can be seen as an extension of the overview article by Liepins, Garfinkel & Kunnathur (1982). All algorithms we discuss are based on the (generalised) Fellegi-Holt paradigm that says that the data of a record should be made to satisfy all edits by changing the fewest possible (weighted) number of fields. The error localisation problem may have several optimal solutions for a record. In contrast to what is common in the literature, most of the algorithms we describe aim to find all optimal solutions rather than just one. As numerical data mostly occur in business surveys, the described algorithms are mainly suitable for business surveys and less so for social surveys. For four algorithms we compare the computing times on six realistic data sets as well as their complexity.

Selective Editing: A Quest for Efficiency and Data Quality

“…According to this paradigm, the data of a record should be made to satisfy all edits by changing the values of the fewest possible number of variables. This paradigm became the standard on which most systems for automatic editing, such as GEIS ( De Waal and Coutinho (2005).…”

The editing of statistical data: methods and techniques for the efficient detection and correction of errors and missing values

“…We refer to Ref 4 or 12 for such a formulation and several solution methods. Alternative references are from Ref 13 where an overview of algorithms for solving the Fellegi–Holt‐based error localization problem for numerical data is presented, and from Ref 14 where an algorithm that solves the error localization problem for a combination of categorical and numerical data is described.…”

Simultaneous Edit and Imputation For Household Data with Structural Zeros

“…This is usually done using a variant of the error localization suggested by Fellegi and Holt (1976), where the values are selected by minimizing the number of fields necessary to turn an erroneous record into a theoretically valid one (Winkler 1995;Winkler and Petkunas 1997). In the imputation step, the values selected in the error localization are replaced with plausibly valid entries (de Waal and Coutinho 2005;de Waal et al 2011). This is usually done by some form of hot deck imputation (Kalton and Kasprzyk 1986;Andridge and Little 2010).…”