A Big Data Framework for Satellite Images Processing using Apache Hadoop and RasterFrames: A Case Study of Surface Water Extraction in Phu Tho, Viet Nam

Nguyen, Dung Viet; Anh, Hong

doi:10.14569/ijacsa.2020.0111289

Cited by 3 publications

(1 citation statement)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We store large-scale remote sensing image data in HDFS to achieve distributed storage and ensure data reliability and high fault tolerance in the data storage layer. In the data loading layer, we use the interfaces in RasterFrames [43] and PyHDFS to interact with HDFS to realize data reading and writing of conventional remote sensing image formats such as tif, png, and jpeg.…”

Section: B Distributed Inference Framework For Object Detection In La...mentioning

confidence: 99%

Object Detection in Large-Scale Remote Sensing Images With a Distributed Deep Learning Framework

Liu

Yan

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

With the accumulation and storage of remote sensing images in various satellite data centers, the rapid detection of objects of interest from large-scale remote sensing images is a current research focus and application requirement. Although some cutting-edge object detection algorithms in remote sensing images perform well in terms of accuracy (mAP), their inference speed is slow and requires high hardware requirements that are not suitable for real-time object detection in large-scale remote sensing images. To address this issue, we propose a fast inference framework for object detection in large-scale remote sensing images. On the one hand, we introduce α-IoU Loss on the YWCSL model to implement adaptive weighted loss and gradient, which achieves 64.62% and 79.54% mAP on DIOR-R and DOTA test sets, respectively. More importantly, the inference speed of the YWCSL model reaches 60.74 FPS on a single NVIDIA GeForce RTX 3080 Ti, which is 2.87 times faster than the current state-of-the-art one-stage detector S 2 A-Net. On the other hand, we build a distributed inference framework to enable fast inference on large-scale remote sensing images. Specifically, we save the images on HDFS for distributed storage and deploy the pretrained YWCSL model on the Spark cluster. In addition, we use a custom partitioner RankPartition to repartition the data to further improve the performance of the cluster. When using 5 nodes, the speedup of the cluster reaches 9.54, which is 90.80% higher than the theoretical linear speedup (5.00). Our distributed inference framework for large-scale remote sensing images significantly reduces the dependence of object detection on expensive hardware resources, which has important research significance for the wide application of object detection in remote sensing images.

show abstract

Section: B Distributed Inference Framework For Object Detection In La...mentioning

confidence: 99%

Object Detection in Large-Scale Remote Sensing Images With a Distributed Deep Learning Framework

Liu

Yan

et al. 2022

IEEE J. Sel. Top. Appl. Earth Observations Remote Sensing

View full text Add to dashboard Cite

show abstract

Computation unit architecture for satellite image processing systems

Pazhani¹

2021

Earth Sci Inform

View full text Add to dashboard Cite

Two-Speed Deep-Learning Ensemble for Classification of Incremental Land-Cover Satellite Image Patches

et al. 2023

View full text Add to dashboard Cite

High-velocity data streams present a challenge to deep learning-based computer vision models due to the resources needed to retrain for new incremental data. This study presents a novel staggered training approach using an ensemble model comprising the following: (i) a resource-intensive high-accuracy vision transformer; and (ii) a fast training, but less accurate, low parameter-count convolutional neural network. The vision transformer provides a scalable and accurate base model. A convolutional neural network (CNN) quickly incorporates new data into the ensemble model. Incremental data are simulated by dividing the very large So2Sat LCZ42 satellite image dataset into four intervals. The CNN is trained every interval and the vision transformer trained every half interval. We call this combination of a complementary ensemble with staggered training a “two-speed” network. The novelty of this approach is in the use of a staggered training schedule that allows the ensemble model to efficiently incorporate new data by retraining the high-speed CNN in advance of the resource-intensive vision transformer, thereby allowing for stable continuous improvement of the ensemble. Additionally, the ensemble models for each data increment out-perform each of the component models, with best accuracy of 65% against a holdout test partition of the RGB version of the So2Sat dataset.

show abstract

A Big Data Framework for Satellite Images Processing using Apache Hadoop and RasterFrames: A Case Study of Surface Water Extraction in Phu Tho, Viet Nam

Cited by 3 publications

References 17 publications

Object Detection in Large-Scale Remote Sensing Images With a Distributed Deep Learning Framework

Object Detection in Large-Scale Remote Sensing Images With a Distributed Deep Learning Framework

Computation unit architecture for satellite image processing systems

Two-Speed Deep-Learning Ensemble for Classification of Incremental Land-Cover Satellite Image Patches

Contact Info

Product

Resources

About