Applying GPU and POSIX thread technologies in massive remote sensing image data processing

Liu, Yuehu; Chen, Bin; Yu, Hao; Zhao, Yong; Huang, Zhou; Fang, Yu

doi:10.1109/geoinformatics.2011.5980671

Cited by 15 publications

(9 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It offers on-line processing of regional to global climate change related quantitative RS data products with these multi-source RS data across data centers. The RS data products generated by MDCPS include vegetation related parameters like NDVI 18 and NPP 19 , radiation and hydrothermal flux related parameters like AOD 20 and SM 21 , as well as global ice change and mineral related parameters. The 5-day global synthetic NDVI parameter product in 2014 generated using MODIS 1km data is showed in figure 10.…”

Section: Experiments and Discussionmentioning

confidence: 99%

pipsCloud: High performance cloud computing for remote sensing big data management and processing

Wang

Yan

et al. 2018

Future Generation Computer Systems

173

View full text Add to dashboard Cite

With the increasing requirement of accurate and up-to-date resource & environmental information for regional and global monitoring, large-region covered multi-temporal, multi-spectral massive remote sensing (RS) datasets are exploited for processing. The remote sensing data processing generally follows a complex multi-stage processing chain, which consists of several independent processing steps subject to types of RS applications. In general the RS data processing for regional environmental and disaster monitoring are recognized as typical both compute-intensive and data-intensive applications.To solve the aforementioned issues efficiently, we propose pipsCloud which combine recent Cloud computing and HPC techniques to enable large-scale RS data processing system as on-demand real-time services. Benefiting from the ubiquity, elasticity and high-level of transparency of Cloud computing model, the massive RS data managing and data processing for dynamic environmental monitoring are all encapsulate as Cloud with Web interfaces. Where, a Hilbert-R + based data indexing mechanism is employed for optimal query and access of RS imageries, RS data products as well as interim data. In the core platform beneath the Cloud services, we provide a parallel file system for massive high-dimensional RS data and offers interfaces for intensive irregular RS data accessing so as to provide improved data locality and optimized I/O performance. Moreover, we adopt an adaptive RS data analysis workflow manage system for on-demand workflow construction and collaborative execution of distributed complex chain of RS data processing, such as forest fire detection, mineral resources and coastline monitoring. Through the experimental analysis we have show the efficiency of the pipsCloud platform.

show abstract

Section: Experiments and Discussionmentioning

confidence: 99%

pipsCloud: High performance cloud computing for remote sensing big data management and processing

Wang

Yan

et al. 2018

Future Generation Computer Systems

173

View full text Add to dashboard Cite

show abstract

“…The total amount of image data at a national level has been more than one petabyte(PB), and the growth rate has exceeded one terabyteeach day. Large amounts of image data need to be stored and processedin real-time [1]. That is difficult in traditional ways, and consumes a lot of time.…”

Section: Introductionmentioning

confidence: 99%

“…With rapid development ofremote sensing and computer technologies, the remote sensing image data obtained by satellite isincreasing dramatically [1]. The total amount of image data at a national level has been more than one petabyte(PB), and the growth rate has exceeded one terabyteeach day.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Massive image data management using HBase and MapReduce

Liu

Chen

Wang

et al. 2013

2013 21st International Conference on Geoinformatics

Self Cite

View full text Add to dashboard Cite

With the rapid development of remote sensing and computer technologies, remote sensing image data obtained by satellite isincreasing dramatically [1]. The speed has exceeded one TB each day and will obviously increase in the future. How to manage it efficiently becomes a problem because traditional waysare expensive and difficultto extend. Hence, we need a scalable and parallel processing model. HBaseand MapReduce meet the needs naturally. In this paper, we propose a method to store massive image data in HBase, and process it using MapReduce. Experimental results illustrate that the speeds of data importing and data processing increase obviously as the cluster of HBase grows.

show abstract

“…In recent years, with the rapid development of surveying and remote sensing technologies, the volume of spatial data has increased dramatically [1][2][3]. Spatial data processing is a typical type of data-intensive applications where users must access and process massive spatial data.…”

Section: Introductionmentioning

confidence: 99%

Hypergraph+: An Improved Hypergraph-Based Task-Scheduling Algorithm for Massive Spatial Data Processing on Master-Slave Platforms

Cheng

Guan

et al. 2016

IJGI

View full text Add to dashboard Cite

Spatial data processing often requires massive datasets, and the task/data scheduling efficiency of these applications has an impact on the overall processing performance. Among the existing scheduling strategies, hypergraph-based algorithms capture the data sharing pattern in a global way and significantly reduce total communication volume. Due to heterogeneous processing platforms, however, single hypergraph partitioning for later scheduling may be not optimal. Moreover, these scheduling algorithms neglect the overlap between task execution and data transfer that could further decrease execution time. In order to address these problems, an extended hypergraph-based task-scheduling algorithm, named Hypergraph+, is proposed for massive spatial data processing. Hypergraph+ improves upon current hypergraph scheduling algorithms in two ways: (1) It takes platform heterogeneity into consideration offering a metric function to evaluate the partitioning quality in order to derive the best task/file schedule; and (2) It can maximize the overlap between communication and computation. The GridSim toolkit was used to evaluate Hypergraph+ in an IDW spatial interpolation application on heterogeneous master-slave platforms. Experiments illustrate that the proposed Hypergraph+ algorithm achieves on average a 43% smaller makespan than the original hypergraph scheduling algorithm but still preserves high scheduling efficiency.

show abstract

Applying GPU and POSIX thread technologies in massive remote sensing image data processing

Cited by 15 publications

References 5 publications

pipsCloud: High performance cloud computing for remote sensing big data management and processing

pipsCloud: High performance cloud computing for remote sensing big data management and processing

Massive image data management using HBase and MapReduce

Hypergraph+: An Improved Hypergraph-Based Task-Scheduling Algorithm for Massive Spatial Data Processing on Master-Slave Platforms

Contact Info

Product

Resources

About