Driving Scientific Applications by Data in Distributed Environments

Saltz, Joel; Çatalyürek, Ümit V.; Kurç, Tahsin; Gray, Mike; Hastings, Shannon; Langella, Steve; Narayanan, S.; Martino, Ryan; Bryant, Steven L.; Peszyńska, Małgorzata; Wheeler, Mary F.; Sussman, Alan; Beynon, Michael D.; Hansen, Christian; Stredney, Don; Sessanna, Dennis

doi:10.1007/3-540-44864-0_37

Cited by 12 publications

(10 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We carried out an experimental performance evaluation of the proposed algorithm using a large oil reservoir simulation dataset [17] generated using an application emulator to control its size and chunking. Table 1 displays the properties of the original dataset and its replicas that were used in the experiments.…”

Section: Resultsmentioning

confidence: 99%

“…In simulation-based oil reservoir management studies [17], the goal is to investigate changes in reservoir characteristics and assess how injection and production wells should be placed to maximize oil production and at the same time minimize effects to the environment. An oil reservoir is a 3-dimensional subsurface volume composed of different types of rocks, soil, holes, and underground water ways.…”

Section: Motivating Applicationmentioning

confidence: 99%

“…Chunks in each replica that intersect the query are categorized as partial or full chunks and into different fragments, and the respective goodness values of the fragments are calculated (steps 2-6). For a given query Q, let us denote the set of all fragments as F and the list of all chosen fragments in decreasing order of goodness value as S. We can apply our greedy search over F (the while loop over steps [8][9][10][11][12][13][14][15][16][17][18]. We choose the fragment with the largest goodness value, move it from F to S, and modify Q by subtracting the range contained by this fragment.…”

Section: Replica Selection Algorithmmentioning

confidence: 99%

See 2 more Smart Citations

Servicing range queries on multidimensional datasets with partial replicas

Weng

Çatalyürek

Kurç

et al. 2005

CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005.

View full text Add to dashboard Cite

Partial replication is one type of optimization to speed up execution of queries submitted to large datasets. In partial replication, a portion of the dataset is extracted, re-organized, and re-distributed across the storage system. The objective is to reduce the volume of I/O and increase I/O parallelism for different types of queries and for the portions of the dataset that are likely to be accessed frequently. When multiple partial replicas of a dataset exist, query execution plan should be generated so as to use the best combination of subsets of partial replicas (and possibly the original dataset) to minimize query execution time.In this paper, we present a compiler and runtime approach for range queries submitted against distributed scientific datasets. A heuristic algorithm is proposed to choose the set of replicas to reduce query execution. We show the efficiency of the proposed method using datasets and queries in oil reservoir simulation studies on a cluster machine.

show abstract

Section: Resultsmentioning

confidence: 99%

Section: Motivating Applicationmentioning

confidence: 99%

Section: Replica Selection Algorithmmentioning

confidence: 99%

See 1 more Smart Citation

Servicing range queries on multidimensional datasets with partial replicas

Weng

Çatalyürek

Kurç

et al. 2005

CCGrid 2005. IEEE International Symposium on Cluster Computing and the Grid, 2005.

View full text Add to dashboard Cite

show abstract

“…Thus, complex numerical reservoir models are needed and it is essential that geological uncertainty be incorporated into these models. An approach is to simulate alternative production strategies (number, type, timing and location of wells) applied to realizations of multiple geostatistical models [9]. Simulations are carried out on a three-dimensional grid.…”

Section: Target Applications and Example Queriesmentioning

confidence: 99%

Supporting SQL-3 Aggregations on Grid-Based Data Repositories

Weng¹,

Agrawal²,

Çatalyürek

et al. 2005

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. There is an increasing trends towards distributed and shared repositories for storing scientific datasets. Developing applications that retrieve and process data from such repositories involves a number of challenges. First, these data repositories store data in complex, low-level layouts, which should be abstracted from application developers. Second, as data repositories are shared resources, part of the computations on the data must be performed at a different set of machines than the ones hosting the data. Third, because of the volume of data and the amount of computations involved, parallel configurations need to be used for both hosting the data and the processing on the retrieved data. In this paper, we describe a system for executing SQL-3 queries over scientific data stored as flatfiles. A relational table-based virtual view is supported on these flat-file datasets. The class of queries we consider involve data retrieval using Select and Where clauses, and processing with user-defined aggregate functions and group-bys. We use a middleware system STORM for providing much of the low-level functionality. Our compiler analyzes the SQL-3 queries and generates many of the functions required by this middleware. Our experimental results show good scalability with respect to the number of nodes as well as the dataset size.

show abstract

“…Data storage is necessary for visualization, snapshots, checkpointing, out-of-core computation, post processing [11], and numerous other reasons. Integrated Parallel Accurate Reservoir Simulation (IPARS) [12] is one such example. IPARS, a software system for large-scale oil reservoir simulation, computes on a three-dimensional data grid at every time step with 9,000 cells.…”

Section: Introductionmentioning

confidence: 99%