High-performance scientific data management system

Abstract

Many scientific applications have large I/O requirements, in terms of both the size of data and the number of files or data sets. Management, storage, efficient access, and analysis of this data present an extremely challenging task. Traditionally, two different solutions have been used for this task: file I/O or databases. File I/O can provide high performance but is tedious to use with large numbers of files and large and complex data sets. Databases can be convenient, flexible, and powerful but do not perform and scale well for parallel supercomputing applications. We have developed a software system, called Scientific Data Manager (SDM), that combines the good features of both file I/O and databases. SDM provides a high-level API to the user and, internally, uses a parallel file system to store real data (using various I/O optimizations available in MPI-IO) and a database to store application-related metadata. In order to support I/O in irregular applications, SDM makes extensive use of MPI-IO's noncontiguous collective I/O functions. Moreover, SDM uses the concept of a history file to optimize the cost of the index distribution using the metadata stored in database. We describe the design and implementation of SDM and present performance results with two regular applications, ASTRO3D and an Euler solver, and with two irregular applications, a CFD code called FUN3D and a Rayleigh-Taylor instability code.

The Office of Science Data-Management Challenge

“…The system must have capabilities for managing metadata generated and derived from scientific data, a query and analysis capability on these metadata, and the ability to manage a large number of files with associated query functions. The metadata management component entails use of a "lightweight" database system (e.g., MDMS [Chou00,No03]). Such a system is responsible for storing and managing metadata and the results of queries.…”

Data Replication Techniques for Data-Intensive Applications

“…We integrated those replication techniques to GEDAS (Grid Environment-based Data Management System ) [6,7] that is a grid toolkit providing a high-level, user-friendly interface to share the remotely produced data among the grid communities.…”

Parallel I/O Scheduling in the Presence of Data Duplication on Multiprogrammed Cluster Computing Systems

“…Although both parallelization and mapping strategies of parallel jobs have improved system utilization and response time to certain extent, obtaining maximum performance of the parallel programs have remained elusive 131. This is because many scientific and engineering applications have large I/O requirements, in terms of both the size of data and the number of files or data sets [20]. Moreover, regardless of the fact that disk technology has been advancing rapidly especially with respect to the storage density, capacity and bandwidth, I/O subsystem continues to be a major bottleneck in many parallel applications [22] 1241 [dl.…”