Big enterprise registration data imputation: Supporting spatiotemporal analysis of industries in China

Abstract

A B S T R A C TBig, fine-grained enterprise registration data that includes time and location information enables us to quantitatively analyze, visualize, and understand the patterns of industries at multiple scales across time and space. However, data quality issues like incompleteness and ambiguity, hinder such analysis and application. These issues become more challenging when the volume of data is immense and constantly growing. High Performance Computing (HPC) frameworks can tackle big data computational issues, but few studies have systematically investigated imputation methods for enterprise registration data in this type of computing environment. In this paper, we propose a big data imputation workflow based on Apache Spark as well as a bare-metal computing cluster, to impute enterprise registration data. We integrated external data sources, employed Natural Language Processing (NLP), and compared several machine-learning methods to address incompleteness and ambiguity problems found in enterprise registration data. Experimental results illustrate the feasibility, efficiency, and scalability of the proposed HPC-based imputation framework, which also provides a reference for other big georeferenced text data processing. Using these imputation results, we visualize and briefly discuss the spatiotemporal distribution of industries in China, demonstrating the potential applications of such data when quality issues are resolved. Only one matchedAddresses with maximum matched degree 1. Chinese word segmentation and address noun selection 2. SQL fuzzy matching 3. Number of matched address records counting 5. Number of matched address records counting Yes Yes Appearance probability: 83% Appearance probability: 17% 7. Output Matched degree: 93.96% Matched degree: 93.96%

Optimizing and accelerating space–time Ripley ’s K function based on Apache Spark for distributed spatiotemporal point pattern analysis

“…The point dataset used in experiments was enterprises registration data in Hubei Province, China (369,826 points in total) from 1949 to 2015 recorded by the bureaus of Administration for Industry and Commerce (AIC) of China. The dataset was cleaned with imputation methods [56], and the study area was administrative boundary of Hubei Province. Spatiotemporal point analysis on these enterprise entities could support evaluation and assessment of industry concentration.…”

Developing Apache Spark Based Ripley’s K Functions for Accelerating Spatiotemporal Point Pattern Analysis

“…The versions of Spark and Hadoop are v2.3 and v2.7 respectively. The experiment datasets were resampled from the enterprises registration data in Chongqing, China (2,119,419 points in total) from year 1949 to year 2018 recorded by the bureaus of Administration for Industry and Commerce (AIC) of China after imputation (Li et at., 2018). For more performance analysis on space-time K function, please refer to our previous research (Wang et al, 2020).…”

Geospatial big data for urban planning and urban management

“…For example, a huge number of Landsat images are utilized in mapping high-resolution global forest cover and the global forest changes in the twentyfirst century are explored (Hansen et al 2013), which is impossible without the support of geospatial big data and the related automatic processing techniques. Based on the huge enterprise registration data in China, the economic and social development situations and trends are revealed by the non-statistic data and novel approaches (Li et al 2018). City-wide fine-grained urban population distribution at building level is achieved by integrating and fusing multisource geospatial big data (Yao et al 2017), which is usually not desired in traditional research.…”