Storage and processing of massive remote sensing images using a novel cloud computing platform

Lin, Feng‐Cheng; Chung, Lan-Kun; Wang, Chunju; Ku, Wen-Yuan; Chou, Tien-Yin

doi:10.1080/15481603.2013.810976

Cited by 21 publications

(8 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of the MapReduce [Dean and Ghemawat 2008] programming model caused a revolution in the processing and management of spatiotemporal data [Guo et al 2017]. MapReduce has facilitated the development of works for data mining of spatiotemporal data [Lin et al 2013;de Assis et al 2017;Song et al 2015;Patterson 2011] and3D objects [Van Den Bergh et al 2012].…”

Section: • 275mentioning

confidence: 99%

SmarT: Machine Learning Approach for Efficient Filtering and Retrieval of Spatial and Temporal Data in Big Data

Oliveira

Rodrigues

Martins

2021

jidm

View full text Add to dashboard Cite

Spatiotemporal data has always been big data. In these days, big data analytics for spatiotemporal data is receiving considerable attention to allow users to analyze huge amounts of data. Traditional big data platforms cannot handle all the challenges of processing spatio-temporal data. Although some big data platforms have been proposed to process a massive volume of spatiotemporal data, neither is considered a clear winner for all possible scenarios. This paper presents the SmarT query engine, a machine learning-based solution that chooses the best big data platform for processing spatiotemporal queries on the fly. In a detailed experimental evaluation, considering the Apache Spark, Elasticsearch, and SciDB big data platforms, the response time decreased up to 22% when using SmarT.

show abstract

Section: • 275mentioning

confidence: 99%

SmarT: Machine Learning Approach for Efficient Filtering and Retrieval of Spatial and Temporal Data in Big Data

Oliveira

Rodrigues

Martins

2021

jidm

View full text Add to dashboard Cite

show abstract

“…For example, Gao et al adapted Hadoop to construct gazetteers from volunteered big geospatial data [12]. Lin et al leveraged Hadoop to store and process massive remote sensing images to support large concurrent user requests [21]. Krishnan et al investigated the use of MapReduce to generate DEM by gridding the LIDAR data [22].…”

Section: Hadoop For Geospatial Data Processingmentioning

confidence: 99%

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Yang

Liu

et al. 2016

IJGI

View full text Add to dashboard Cite

Efficient processing of big geospatial data is crucial for tackling global and regional challenges such as climate change and natural disasters, but it is challenging not only due to the massive data volume but also due to the intrinsic complexity and high dimensions of the geospatial datasets. While traditional computing infrastructure does not scale well with the rapidly increasing data volume, Hadoop has attracted increasing attention in geoscience communities for handling big geospatial data. Recently, many studies were carried out to investigate adopting Hadoop for processing big geospatial data, but how to adjust the computing resources to efficiently handle the dynamic geoprocessing workload was barely explored. To bridge this gap, we propose a novel framework to automatically scale the Hadoop cluster in the cloud environment to allocate the right amount of computing resources based on the dynamic geoprocessing workload. The framework and auto-scaling algorithms are introduced, and a prototype system was developed to demonstrate the feasibility and efficiency of the proposed scaling mechanism using Digital Elevation Model (DEM) interpolation as an example. Experimental results show that this auto-scaling framework could (1) significantly reduce the computing resource utilization (by 80% in our example) while delivering similar performance as a full-powered cluster; and (2) effectively handle the spike processing workload by automatically increasing the computing resources to ensure the processing is finished within an acceptable time. Such an auto-scaling approach provides a valuable reference to optimize the performance of geospatial applications to address data-and computational-intensity challenges in GIScience in a more cost-efficient manner.

show abstract

“…Based on Cloud computing technologies, Chen et al (2008) built a high performance workflow system MRGIS using MapReduce clusters to execute GIS applications efficiently; Park et al (2010) used Hadoop HDFS and MapReduce to do massively parallel processing of 3D GIS data, they found the computing time is vastly reduced with a cluster of computing nodes. In GIS PaaS application, Gong et al (2010) proposed to integrate GIS geoprocessing functions with scalable Microsoft Cloud computing platform Azure for providing geoprocessing capabilities; Aji et al (2013) presented Hadoop GIS for running large scale spatial queries, it's a scalable and high performance spatial data warehousing system; Lin et al (2013) proposed and implemented an architectural design for a novel Cloud computing platform based on two Web Coverage Service and Web Map Service interfaces from the Open Geospatial Consortium (OGC), cloud storage from Hadoop HDFS, and image processing from MapReduce; Gao et al (2014) built a scalable distributed platform and a high performance geoprocessing workflow based on the Hadoop ecosystem to harvest crowd-sourced gazetteer entries.…”

Section: Overview Of Big Data and Cloud Computing Cloudgis Technologiesmentioning

confidence: 99%

Building Spatiotemporal Cloud Platform for Supporting Gis Application

Song

Jin²,

et al. 2015

ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci.

View full text Add to dashboard Cite

ABSTRACT:Traditional geospatial information platforms are built, managed and maintained by the geoinformation agencies. They integrate various geospatial data (such as DLG, DOM, DEM, gazetteers, and thematic data) to provide data analysis services for supporting government decision making. In the era of big data, it is challenging to address the data-and computing-intensive issues by traditional platforms. In this research, we propose to build a spatiotemporal cloud platform, which uses HDFS for managing image data, and MapReduce-based computing service and workflow for high performance geospatial analysis, as well as optimizing autoscaling algorithms for Web client users' quick access and visualization. Finally, we demonstrate the feasibility by several GIS application cases.

show abstract

Storage and processing of massive remote sensing images using a novel cloud computing platform

Cited by 21 publications

References 10 publications

SmarT: Machine Learning Approach for Efficient Filtering and Retrieval of Spatial and Temporal Data in Big Data

SmarT: Machine Learning Approach for Efficient Filtering and Retrieval of Spatial and Temporal Data in Big Data

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Building Spatiotemporal Cloud Platform for Supporting Gis Application

Contact Info

Product

Resources

About