Dynamic data science and official statistics

Abstract

Many of the challenges and opportunities of data science have to do with dynamic factors: a growing volume of administrative and commercial data on individuals and establishments, continuous flows of data and the capacity to analyze and summarize them in real time, and the necessity for resources to maintain them. With its emphasis on data quality and supportable results, the practice of Official Statistics faces a variety of statistical and data science issues. This article discusses the importance of population frames and their maintenance; the potential for use of multi‐frame methods and linkages; how the use of large scale non‐survey data may shape the objects of inference; the complexity of models for large data sets; the importance of recursive methods and regularization; and the benefits of sophisticated spatial visualization tools in capturing spatial variation and temporal change. The Canadian Journal of Statistics 46: 10–23; 2018 © 2017 Statistical Society of Canada

Data science training for official statistics: A new scientific paradigm of information and knowledge development in national statistical systems

“…The growing volume of administrative data, granular data and registered based statistics with continuous flows are an attractive opportunity for NSOs to produce real-time quality statistics delivering on expectations of public institutions, the business community and the public at a lower cost when compared with traditional methods [13]. For this purpose, data science offers new approaches to process data systematically applying both traditional statistical techniques and new data mining and machine learning methodologies to describe and illustrate, condense, and evaluate data.…”

Planejamento e Políticas Públicas

“…Aspectos como mínima granularidade, mencionado por Nasem (2017b) e Gurwitz ( 2012), e usabilidade, citado por Unece (2015), podem ser parte da dimensão relevância. Processos de gestão e governança de dados, com integração, linkagem, limpeza, processamento, ponderação e imputação de dados (Vaju e Meszaros, 2018;Thompson, 2018;Máslankowski e Nowicka, 2018;Lauro e Traverso, 2018;Hand, 2018;Struijs e Daas, 2014;Nasem, 2017b;Biffignandi e Signorelli, 2015;Unece, 2015;Uluwiyah, 2016) podem fazer parte das dimensões extração, interpretação e transformação. Variabilidade, volatilidade, instabilidade e disrupção das fontes (Silva, 2016;Biffignandi e Signorelli, 2015;Hajnovic, 2018) entram em sustentabilidade da fonte, em linha com mudanças metodológicas que podem acontecer (Hand, 2018) e culminar na perda da comparabilidade temporal.…”

Ratio Edits Based on Statistical Tolerance Intervals

“…Some texts dedicated to the general topic of outlier detection include Barnett and Lewis (1994), Rousseeuw and Leroy (2003), and Aggarwal (2013). A number of outlier detection methods are also available in the literature and can be used for developing ratio edit tolerances; we refer to Thompson and Sigman (1999) and Rais (2008) for a review and comparison of these methods as they apply to the ratio edit problem. Thompson and Sigman (1999) compared different methods for generating ratio edit tolerances, which focused on "Type I" and "Type II" errors.…”

“…A number of outlier detection methods are also available in the literature and can be used for developing ratio edit tolerances; we refer to Thompson and Sigman (1999) and Rais (2008) for a review and comparison of these methods as they apply to the ratio edit problem. Thompson and Sigman (1999) compared different methods for generating ratio edit tolerances, which focused on "Type I" and "Type II" errors. A Type I error flags a ratio value as inconsistent or wrong when it is not so.…”

“…A Type II error flags an inconsistent ratio as consistent or correct. Thompson and Sigman (1999) recommended a stepwise approach for developing ratio edit tolerances, while Thompson and Adeshiyan (2003) discussed the effects of ratio edit and imputation procedures on data quality for the 1997 Economic Census. Both articles also emphasized the importance of incorporating subject-matter expertise when developing the ratio edits.…”