An Efficient Organization Method for Large-Scale and Long Time-Series Remote Sensing Data in a Cloud Computing Environment

Abstract

Historical Earth observation (EO) data have played an important role in long-term scientific and environmental monitoring. The effective organization of large-scale and longterm remote-sensing data to achieve efficient retrieval and access has become one of the important issues. However, inherent big data characteristics, such as a large scale, and asymmetrical temporal and spatial distributions, have caused problems with the efficiency of data retrieval and access. Therefore, this study proposes an efficient data organization method for use in a cloudcomputing environment that has two aims. First, it addresses the problem of low retrieval efficiency. An asymmetrical index model for the image metadata is constructed that is based on a unified spatio-temporal grid coding; a pre-partitioning mechanism under the HBase architecture is established to realize the uniform storage of the metadata with an asymmetrical spatiotemporal distribution and to avoid retrieval efficiency bottlenecks caused by a load imbalance. Second, it addresses low access efficiency. By dividing the remote-sensing image into tiles, a unified spatiotemporal code is established for each tile, and a consistent hash operation is performed; tiles with similar hash values are stored in the same or adjacent HDFS (Hadoop Distributed File System) nodes. In this way, tiles with temporal or spatial correlations can be gathered and stored, and lots of disk seeks can be avoided during retrieval, thereby greatly improving the data access efficiency. Comparative experiments showed that the data organization method can effectively improve the retrieval and access efficiencies of large-scale and long time-series remotesensing data in a cloud-computing environment.