Big Data, Official Statistics and Some Initiatives by the Australian Bureau of Statistics

Abstract

Summary Official statisticians have been dealing with a diversity of data sources for decades. However, new sources of data in the Big Data domain provide an opportunity to deliver a more efficient and effective statistical service. This paper outlines a number of considerations for the official statistician when deciding whether to embrace a particular new data source in the regular production of official statistics. The principal considerations are relevance, business benefit, and the validity of using the source for official statistics in finite population inferences or analytic inferences. The paper also describes the Big Data Flagship Project of the Australian Bureau of Statistics (ABS), which has been established to provide the opportunity for the ABS to gain practical experience in assessing the business, statistical, technical, computational and other issues in using Big Data. In addition, ABS participation in national and international activities in this area will help it share experience and knowledge, while collaboration with academics will enable ABS to better acquire the capability to address business problems using the new sources of data as part of the solution.

The opportunities, challenges and risks of big data for official statistics

“…A first issue is to gain access to the required big data in the first place for assessment, experimenting, trialing and adoption [10,12,28]. Although some big data are produced by public agencies, such as weather data, some website and administrative systems, and some transport data, much big data are presently generated by private companies such as mobile phone, social media, utility, financial, credit, insurance and retail companies [16].…”

The production of salary profiles of ICT professionals: Moving from structured database to big data analytics

“…The official statistical community, who have been dealing with a diversity of data sources for decades, are incrementally exploring whether to embrace big data sources in the regular production of official statistics [40]. Succinctly put, the principal considerations that warrant due attention are relevance, validity of using the source especially those originate from private sectors, methodological and analytics procedures and inferences as well as business benefit [40].…”

“…Succinctly put, the principal considerations that warrant due attention are relevance, validity of using the source especially those originate from private sectors, methodological and analytics procedures and inferences as well as business benefit [40]. Despite challenges, the official statisticians are increasingly acknowledging big data sources have the potential to facilitate the linkage between economic statistics and the people that they are about, improving quality and granularity of existing data and availability of new data that cannot be collected through traditional means [42].…”

“…Data quality and statistical integrity have been an age old concern in any measurement procedure and quantitative analysis [3,12,40] and BDA is not an exception. Issues of concerns in BDA authentication include exactitude, genuineness, originality, legitimacy, legality, rightfulness, dependability, authoritativeness, credibility, factualness, trustworthiness, historicity and bona fides.…”

“…Government organisations including official statistical systems are also no exception to BDA phenomena, notions and trends. Specifically, Big Data and Official Statistics initiatives by Eurostat and Department of Statistics of Hong Kong, Official Statistics Australian Bureau of Statistics (ABS) Big Data Flagship and Irish Centre for High-End Computing (ICHEC) are already experimenting data residing in web, e-commerce and sensors using BDA approach [40,42].…”