SPDO: High-throughput road distance computations on Spark using Distance Oracles

Peng, Shangfu; Sankaranarayanan, Jagan; Samet, Hanan

doi:10.1109/icde.2016.7498328

Cited by 10 publications

(12 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The covering algorithm implemented on Spark is illustrated in stages 1 and 2 in Figure 3. The distributed files are read from Hadoop Distributed File System (HDFS) and transformed into IndexedRDD [29]. The parallel covering process in stage 1 consists of many covering processes.…”

Section: Stage 3 Is the Lloyd Iteration Phase In Which Lloyd Iteratimentioning

confidence: 99%

Self‐Adaptive K‐Means Based on a Covering Algorithm

et al. 2018

View full text Add to dashboard Cite

The K-means algorithm is one of the ten classic algorithms in the area of data mining and has been studied by researchers in numerous fields for a long time. However, the value of the clustering number k in the K-means algorithm is not always easy to be determined, and the selection of the initial centers is vulnerable to outliers. This paper proposes an improved K-means clustering algorithm called the covering K-means algorithm (C-K-means). The C-K-means algorithm can not only acquire efficient and accurate clustering results but also self-adaptively provide a reasonable numbers of clusters based on the data features. It includes two phases: the initialization of the covering algorithm (CA) and the Lloyd iteration of the K-means. The first phase executes the CA. CA self-organizes and recognizes the number of clusters k based on the similarities in the data, and it requires neither the number of clusters to be prespecified nor the initial centers to be manually selected. Therefore, it has a “blind” feature, that is, k is not preselected. The second phase performs the Lloyd iteration based on the results of the first phase. The C-K-means algorithm combines the advantages of CA and K-means. Experiments are carried out on the Spark platform, and the results verify the good scalability of the C-K-means algorithm. This algorithm can effectively solve the problem of large-scale data clustering. Extensive experiments on real data sets show that the accuracy and efficiency of the C-K-means algorithm outperforms the existing algorithms under both sequential and parallel conditions.

show abstract

Section: Stage 3 Is the Lloyd Iteration Phase In Which Lloyd Iteratimentioning

confidence: 99%

Self‐Adaptive K‐Means Based on a Covering Algorithm

et al. 2018

View full text Add to dashboard Cite

show abstract

“…Details would be explained in Section 3.2. SPDO [29] is another method of using ϵ-DO inside a distributed key value store, e.g., Apache Spark. Another databasecentric method is HLDB [11] which can answer exact network distance queries and full shortest-path even for an area as large as Europe containing 18 million vertices with complex SQL queries.…”

Section: Related Workmentioning

confidence: 99%

“…Our previous work [27][28][29] discussed how to utilize DO (G) in different queries and settings. In SPDO [29], we proposed a general distributed framework to achieve a high throughput relying on the distance oracle of the whole USA road network.…”

Section: Dos Frameworkmentioning

confidence: 99%

“…Clearly, using the geodesic or Euclidean distance to approximate network distance can produce significant errors. To measure how big the difference is, we tabulated ratios of network distance to geodesic distance in some regions in Figure 2, termed the route directness spectrum (RDS) [29]. It shows the distortion in approximating network distance with geodesic distance.…”

Section: Introductionmentioning

confidence: 99%

“…These two papers introduce the distance oracle, but is not scalable to road networks with more than 80, 000 vertices. (2) Our previous work SPDO [29] discussed how to obtain high throughput performance using ϵ-DO in a distributed key-value store such as Apache Spark for spatial analysis on the continental road networks such as the entire USA. (3) Our previous demo work CDO [28] claimed that the reaction of some companies, such as UPS, Uber, PlaceIQ, etc, was that typical queries are concentrated in a small local region rather than the whole continental region, termed the spatial concentration property.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Dos

Peng

Sankaranarayanan

Samet

2018

Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems

Self Cite

View full text Add to dashboard Cite

Large analytic applications on road networks including simulations, logistics, location-based advertisement, and transportation planning require shortest distance/time methods that provide high throughput (i.e., distance/time computations per second). Our previous work discussed how to process graph distance computations in a PostgreSQL database on a large road network, e.g., 60K distance computations per second per machine, how to "scale out" by using a Spark cluster to achieve 73.8K distance computations per second per machine, and how to obtain a extremely high-throughput solution in memory for city-sized road networks, e.g., 6.7M distance computations per second. However, there is no solution that could achieve more than 1M throughput for large road networks. In an industrial setting, most state-of-the-art solutions yield 5K − 10K shortest distance computations per second per machine even with multi-threads. In this paper, we propose a new distance oracle system (DOS) for large road networks. It can solve most spatial analytic queries, and its throughput achieves 5M distance computations per second even on the whole USA road network. For example, a 10K × 10K origin-distance (OD) matrix can be computed in 20 seconds.

show abstract