Summary Official statisticians have been dealing with a diversity of data sources for decades. However, new sources of data in the Big Data domain provide an opportunity to deliver a more efficient and effective statistical service. This paper outlines a number of considerations for the official statistician when deciding whether to embrace a particular new data source in the regular production of official statistics. The principal considerations are relevance, business benefit, and the validity of using the source for official statistics in finite population inferences or analytic inferences. The paper also describes the Big Data Flagship Project of the Australian Bureau of Statistics (ABS), which has been established to provide the opportunity for the ABS to gain practical experience in assessing the business, statistical, technical, computational and other issues in using Big Data. In addition, ABS participation in national and international activities in this area will help it share experience and knowledge, while collaboration with academics will enable ABS to better acquire the capability to address business problems using the new sources of data as part of the solution.

No abstract

Summary The statistical challenges in using big data for making valid statistical inference in the finite population have been well documented in literature. These challenges are due primarily to statistical bias arising from under‐coverage in the big data source to represent the population of interest and measurement errors in the variables available in the data set. By stratifying the population into a big data stratum and a missing data stratum, we can estimate the missing data stratum by using a fully responding probability sample and hence the population as a whole by using a data integration estimator. By expressing the data integration estimator as a regression estimator, we can handle measurement errors in the variables in big data and also in the probability sample. We also propose a fully nonparametric classification method for identifying the overlapping units and develop a bias‐corrected data integration estimator under misclassification errors. Finally, we develop a two‐step regression data integration estimator to deal with measurement errors in the probability sample. An advantage of the approach advocated in this paper is that we do not have to make unrealistic missing‐at‐random assumptions for the methods to work. The proposed method is applied to the real data example using 2015–2016 Australian Agricultural Census data.

No abstract