Global static indexing for real-time exploration of very large regular grids

Abstract

In this paper we introduce a new indexing scheme for progressive traversal and visualization of large regular grids. We demonstrate the potential of our approach by providing a tool that displays at interactive rates planar slices of scalar field data with very modest computing resources. We obtain unprecedented results both in terms of absolute performance and, more importantly, in terms of scalability. On a laptop computer we provide real time interaction with a 2048 3 grid (8 Giga-nodes) using only 20MB of memory. On an SGI Onyx we slice interactively an 8192 3 grid (½ tera-nodes) using only 60MB of memory. The scheme relies simply on the determination of an appropriate reordering of the rectilinear grid data and a progressive construction of the output slice. The reordering minimizes the amount of I/O performed during the out-of-core computation. The progressive and asynchronous computation of the output provides flexible quality/speed tradeoffs and a timecritical and interruptible user interface.

Fast Multiresolution Reads of Massive Simulation Datasets

“…The format provides efficient, cache oblivious, progressive access to data by utilizing a hierarchical Z (HZ) order for storage [20]. The HZ order is calculated for each data sample using the spatial coordinates of that sample.…”

“…-interactive view-dependent visualization of massive datasets using VisIt -efficient parallel reads of complete checkpoint data for simulation restarts -parallel multiresolution reads for summary analysis In order to address both parallel full resolution reads, as well as parallel or serial partial resolution reads we utilize the IDX data format [20]. IDX is a hierarchical multiresolution data format with support for both lightweight serial and fast distributed parallel I/O.…”

Remote and Distributed Visualization Arc hitectures

“…One form is to optimize the use of distributed network data caches and replicas, so a request for data goes to a "nearby" rather than "distant" source [1,30], as well as to leverage high performance protocols that are more suitable than a TCP for bulk data transfers [1]. Other optimizations leverage data subsetting, filtering, and progressive transmission from remote sources to reduce the amount of data payload crossing the network [11,23]. Some systems, like the Distributed Parallel Storage System (DPSS), provide a scalable, high performance, distributed-parallel data storage system that can be optimized for data access patterns and the characteristics of the underlying network [31].…”

Dynamic Chunking for Out-of-Core Volume Visualization Applications

“…Pascucci and Frank [10] describe a scheme to reorganizes regular grid data in a way that leads to efficient disk access and enables extracting multiple resolution versions of the data.…”