Large interactive visualization of density functions on big data infrastructure

Abstract

Point set visualization is required in lots of visualization techniques. Scatter plots as well as geographic heat-maps are straightforward examples. Data analysts are now well trained to use such visualization techniques. The availability of larger and larger datasets raises the need to make these techniques scale as fast as the data grows. The Big Data Infrastructure offers the possibility to scale horizontally. Designing point set visualization methods that fit into that new paradigm is thus a crucial challenge. In this paper, we present a complete architecture which fully fits into the Big Data paradigm and so enables interactive visualization of heatmaps at ultra-scale. A new distributed algorithm for multi-scale aggregation of point set is given and an adaptive GPU based method for kernel density estimation is proposed. A complete prototype working with Hadoop, HBase, Spark and WebGL has been implemented. We give a benchmark of our solution on a dataset having more than 2 billion points.

HeatPipe: High Throughput, Low Latency Big Data Heatmap with Spark Streaming

“…As described above, Perrot et al [15] have proposed an efficient solution to implement a batch layer for heatmap visualization. Their solution uses a multilevel aggregation of the data to store weighted abstract point set with increasing levels-of-detail into the serving layer (i.e.…”

“…In the following, we propose a speed layer that can be used together with the Perrot et al [15] batch layer to build a lambda architecture which enables realtime exploration experience on streamed data. More precisely, the speed layer will maintain a partial abstract pyramid which will be merged on the fly with the batch layer pyramid by the serving layer.…”

“…Given an aggregation distance d, it guarantees that no point is further than d from a representative and no two representative are closer than d. Furthermore, it guarantees a bounded maximal number of representative in relation with the chosen d, as shown in [15].…”

“…In this article, we expand on the LIViD batch solution [15] to create an algorithm for streaming multilevel heatmap computation. First, we present the state of the art in density visualization and big data processing frameworks.…”

“…This is not possible with Big Data, since the whole dataset might not fit on the client computer or may require too much time to be transferred through the network on a massively parallel architecture. A solution was recently proposed by Perrot et al [15]. They designed a system which enables interactive heatmap visualization of billions of points on big data infrastructure.…”