Parallel Processing of Big Point Clouds Using Z-Order-Based  Partitioning

Alis, Christian M.; Boehm, J.; Liu, K.

doi:10.5194/isprs-archives-xli-b2-71-2016

Cited by 6 publications

(4 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The experimental results showed that Apache Spark had a better performance than traditional DBMSs for various types of geospatial queries [26]. The methods based on Apache Spark for large point cloud management are described in [29,30]. The method for ingesting the point clouds in the Apache Spark data structures is presented in [29].…”

Section: (A)mentioning

confidence: 99%

See 1 more Smart Citation

Big Data in Smart City: Management Challenges

et al. 2021

View full text Add to dashboard Cite

Smart cities use digital technologies such as cloud computing, Internet of Things, or open data in order to overcome limitations of traditional representation and exchange of geospatial data. This concept ensures a significant increase in the use of data to establish new services that contribute to better sustainable development and monitoring of all phenomena that occur in urban areas. The use of the modern geoinformation technologies, such as sensors for collecting different geospatial and related data, requires adequate storage options for further data analysis. In this paper, we suggest the biG dAta sMart cIty maNagEment SyStem (GAMINESS) that is based on the Apache Spark big data framework. The model of the GAMINESS management system is based on the principles of the big data modeling, which differs greatly from standard databases. This approach provides the ability to store and manage huge amounts of structured, semi-structured, and unstructured data in real time. System performance is increasing to a higher level by using the process parallelization explained through the five V principles of the big data paradigm. The existing solutions based on the five V principles are focused only on the data visualization, not the data themselves. Such solutions are often limited by different storage mechanisms and by the ability to perform complex analyses on large amounts of data with expected performance. The GAMINESS management system overcomes these disadvantages by conversion of smart city data to a big data structure without limitations related to data formats or use standards. The suggested model contains two components: a geospatial component and a sensor component that are based on the CityGML and the SensorThings standards. The developed model has the ability to exchange data regardless of the used standard or the data format into proposed Apache Spark data framework schema. The verification of the proposed model is done within the case study for the part of the city of Novi Sad.

show abstract

Section: (A)mentioning

confidence: 99%

“…The method for ingesting the point clouds in the Apache Spark data structures is presented in [29]. The indexing of point clouds based on space filling curves is presented in [30]. The methods for classification, feature identification, and change detection using large point clouds are described in [26,27,29,31,32].…”

Section: (A)mentioning

confidence: 99%

Big Data in Smart City: Management Challenges

et al. 2021

View full text Add to dashboard Cite

show abstract

“…In addition, Spark provides a richer interface so that applications need not follow the rigid structure of one map and one reduce function as per the original MapReduce framework. The use of Spark for LiDAR point cloud analysis is seen in various research (e.g., [16,18,[37][38][39][40]). In all of those cases, parallel computing was used for post-acquisition data analysis.…”

Section: Parallel Computing For Lidar Data Analysismentioning

confidence: 99%

Optimizing Urban LiDAR Flight Path Planning Using a Genetic Algorithm and a Dual Parallel Computing Framework

Laefer

Byrne

2021

Remote Sensing

View full text Add to dashboard Cite

This paper introduces a genetic algorithm (GA) and a beam tracing algorithm incorporated within a dual parallel computing framework to optimize urban aerial laser scanning (ALS) missions to maximize vertical façade data capture, as needed for many three-dimensional reconstruction and modeling workflows. The optimization employs a low-density point cloud from the site of interest as a spatial representation of the urban scene. The GA is suitable for LiDAR flight path optimization due to its capability of handling open-ended problems that have many solutions. However, GAs require evaluating a very large number of candidates. The use of an initial point cloud allows realistic modeling of the urban environment in the optimization at the cost of high data input volumes. To cope with the computational and data demands, a dual parallel computing framework was devised. The parallel computing framework consists of two layers of parallelization. In the upper layer, multiple evaluators work in parallel and in conjunction with a main multi-threading GA optimizer to perform GA operations and evaluate the flight paths. In the lower layer, to evaluate assigned flight paths, each evaluator distributes its data and computation to multiple executors, which can reside on multiple physical nodes of a distributed-memory computing cluster. In addition to parallelism, the data partitioning on the lower layer allows out-of-core computation. Namely, data partitions are efficiently transferred between disks and memory so that only relevant subsets of data are kept in the main memory. The objective of the proposed method is threefold: (1) search for flight paths that yield the highest numbers of vertical points, (2) create a means to explicitly consider the detailed spatial configuration of urban environments, and (3) assure that the proposed optimization strategy is fast and can scale to large problem sizes. Multiple experiments were conducted and demonstrated the success of the proposed method. Converged results were achieved after dozens of generations within two hours. Two flight paths identified by the GA as the most and the least optimal candidates were deployed in real flight missions. The optimal flight path captured 16% more vertical points than the least optimal one, slightly higher than the 13% predicted. Both layers of parallelization were efficient: 13.1/16 for the lower layer and 3.2/4 for the upper layer. The two complementary layers of parallelization allowed flexible and efficient use of distributed computing resources to reduce the runtime. The scalability of the proposed approach was successfully demonstrated up to a data size of 460 million points. The optimization results were realistic and aligned well with the test flight results.

show abstract

“…We have already shown the successful use of cloud based machine learning for point cloud classification (Liu and Boehm, 2015). While the established tools do not typically provide spatial functionality such as indexing an query, these can be implemented on top of existing frameworks (Alis et al, 2016).…”

Section: Concuisons and Outlookmentioning

confidence: 99%

Sideloading – Ingestion of Large Point Clouds Into the Apache Spark Big Data Engine

Boehm¹,

Liu²,

Alis³

2016

Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.

Self Cite

View full text Add to dashboard Cite

ABSTRACT:In the geospatial domain we have now reached the point where data volumes we handle have clearly grown beyond the capacity of most desktop computers. This is particularly true in the area of point cloud processing. It is therefore naturally lucrative to explore established big data frameworks for big geospatial data. The very first hurdle is the import of geospatial data into big data frameworks, commonly referred to as data ingestion. Geospatial data is typically encoded in specialised binary file formats, which are not naturally supported by the existing big data frameworks. Instead such file formats are supported by software libraries that are restricted to single CPU execution. We present an approach that allows the use of existing point cloud file format libraries on the Apache Spark big data framework. We demonstrate the ingestion of large volumes of point cloud data into a compute cluster. The approach uses a map function to distribute the data ingestion across the nodes of a cluster. We test the capabilities of the proposed method to load billions of points into a commodity hardware compute cluster and we discuss the implications on scalability and performance. The performance is benchmarked against an existing native Apache Spark data import implementation.

show abstract

Parallel Processing of Big Point Clouds Using Z-Order-Based Partitioning

Cited by 6 publications

References 8 publications

Big Data in Smart City: Management Challenges

Big Data in Smart City: Management Challenges

Optimizing Urban LiDAR Flight Path Planning Using a Genetic Algorithm and a Dual Parallel Computing Framework

Sideloading – Ingestion of Large Point Clouds Into the Apache Spark Big Data Engine

Contact Info

Product

Resources

About